I really doubt that anything close to a majority of your followers are from spam accounts. What you are experiencing is probably just a natural ceiling. There are some old accounts with more than 50K followers, but not many.
I'm not expert on acquiring Medium followers. It's taken me three+ years to get to 3K, but I also spend a lot of my time in the fiction world, and the occasional software project that comes up, so I'm actually pretty happy with that.
That said, I am very sure that if I spent 24/7 on Medium, I wouldn't acquire 50,000 followers. You have a talent there, for sure.
I have a Ghost account, too. I started on Substack, but then I was alerted by some folks that they've decided it's okay to host Nazi and other flagrantly racist accounts, so I switched to Ghost, which has a broader commitment against that kind of stuff and is basically the same thing.
I use that for my fiction, mostly. A mix is good to have, I think. We don't want to put all our eggs in one basket.
Good job! I remember when I first followed you you were pretty new here. It's been fun to watch your growth.