I don't think using the accept header is a good idea.
The main reason is that it doesn't necessarily play well with proxies. Most notoriously, Vary: Accept is not supported by Cloudflare (with the exception of images). This means it's not safe to use the accept header if your site will ever be backed by Cloudflare, as you could end up returning cached HTML to a JSON client or vice-versa.
https://developers.cloudflare.com/cache/concepts/cache-contr...
My other problem with it is that it's not very discoverable - it can be surprising to interact with an endpoint that varies based on the accept header. It's not commonly used, so many developers are unfamiliar with it.
And even if you do understand it, it's still hard to tell which endpoints support Vary and which don't.
Finally: I like to debug things by copying and pasting URLs around. The Accept header doesn't support that kind of communication - you need to share not just the URL but also the accept header you were using when you saw a specific response.