undefined | Better HN

Skip to content

Top New Best Ask Show Jobs

undefined | Better HN

0 pointsllmtosser7mo ago0 comments

This is not true.

No inference engine does all of:

- Model switching

- Unload after idle

- Dynamic layer offload to CPU to avoid OOM

0 comments

ekianjo7mo ago

this can be added to llama.cpp with llama.swap currently so even without Ollama you are not far off

j / k navigate · click thread line to collapse