Almost 100% of the time when somebody says “curious what you recommend instead of _____” the best answer is “write a program”, but for some reason programmers aren’t supposed to do that any more.

That's a question that has no single answer. Spark is a giant omnibus project with all sorts of bells and whistles, and the viable alternatives are going to vary wildly, depending on which subset of them you actually need/use.

If you want a single, simple answer that's easy to sell to management, I'd refer you back to the last paragraph of my original post.

I'm not a user myself, but just from watching the chatter in the world it seems dask and rapids are quickly overtaking spark in mindshare, or at least enthusiasm, and that spark is following hadoop/mapreduce to the Big Data Graveyard.