In-browser LLM inference engine with WebGPU and OpenAI API (opens in new tab)

Comes with ability to do full structured generation with json schema

also a in-browser demo https://chat.webllm.ai/

So cool. No more privacy concerns!

It's good to see that it contains the standard OpenAI API interface in JavaScript which looks very convenient.

AMAZING!