Out of that frustration I've built AnyLM, which aims to solve that by providing a single, native application (currently for Windows, with macOS and more planned) to interact with your local models from LM Studio/Ollama but also models from api providers like OpenAI, Anthropic and Google.
This has been a fun (and challenging!) project. I'd be super grateful for any feedback, suggestions, or if you just want to try it out and let me know what you think!