I tend to build my prototypes as single file Elixir scripts. These are sometimes rather bulky with a bunch of logic and IO modules with some helper modules on the side. It's only when I believe in the idea that I reach for phx_new and start writing tests and copying in code into a full Phoenix project.
When I started out with Phoenix I learned a lot from studying the boilerplate it can generate. If you just need a route and serve some JSON or a file on it you can turn off most of the code generation and do precisely that, in case you're sure that's what you want. I think the default being that you can immediately flesh out auth and CRUD for whatever tabular data model you want is rather sensible.