I suspect getting nodes to connect across the cluster is easy in most stacks nowadays, kubernetes or not.
But the idea that you can just call Module.func(), and not care WHICH node it runs in, right out of the box, is quite nice.
I would wager though that 90% of elixir/phoenix devs just aren't doing any of that most of the time, instead just using it like a more performant Ruby on Rails, with ecto being awesome too. :)