Yeah, dealing with auth can be painful. The adonis provided auth and bouncer do a decent job of abstracting some of that away, but like most js implementations, it's got rough edges (the docs and vids really help).
I'm not familiar with Elixir/Phoenix, but from reading other recommendations in this thread and after having a quick look, I need to take a closer look :-)