Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI (opens in new tab)

What are a few use-cases you want to see this used for?

Thanks! It means a lot to hear you say that.

I am kind of militant about this. The ability to run great AI models locally is critical, not just for this sector, but for innovation overall. The bar of "build a few datacenters" is far too high for all but the largest countries and companies in the world.

100%. We don't know what's going to happen in the future. Things are evolving so quickly. Hopefully pushing back on centralization now will keep the ecosystem healthier and give developers real options outside the big two/three cloud providers.

I think that enterprises and small businesses alike need stuff like this, regardless of whether they're software companies or some other vertical like healthcare or legal. I worked at IBM for over a decade and it was always preferable to start with an open source framework if it fit your problem space, especially for internal stuff. We shipped products with components built on Elastic, Drupal, Express, etc.

You could make the same argument for Kubernetes. If you have the cash and the team, why not build it yourself? Most don't have the expertise or the time to find/train the people who do.

People want AI that works out of the box on day one. Not day 100.

Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience.

Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normally have deep pockets).

What do you think is a good starting place?

I hear you and valid. A mixture of Models is probably a better phrase - we are constantly moving between AI experts and very skilled Developers who use OpenAI endpoints and call it AI, so we are constantly working on finding the correct language. This was a miss though - will do better :)

Appreciate that — and totally agree. The “who cares / who pays” question is exactly why this hasn’t scaled before.

Our bet is that the timing’s finally right: local inference, smaller and more powerful open models (Qwen, Granite, Deepseek), and enterprise appetite for control have all converged. We’re working with large enterprises (especially in regulated industries) where innovation teams need to build and run AI systems internally, across mixed or disconnected environments.

That’s the wedge — not another SaaS, but a reproducible, ownable AI layer that can actually move between cloud, edge, and air-gapped. Just reach out, no intro needed - robert @ llamafarm.dev

Yes, our goal is to provide a stable, open source platform on top of the cutting-edge AI tools. We can systematically update dependencies as needed and ensure that outputs meet quality requirements.

We also have plans for eval features in the product so that users can measure the quality of changes over time, whether to their own project configs or actual LlamaFarm updates.

Yes, all that's a bit hand-wavy, I know. :-) But we do recognize the problem and have real ideas on solutions. But execution is everything. ;-)

Multiple areas of degradation. Typically, you don't ship a dataset to prod and then never change it. You want the system to continue to learn and improve as new data is available. This can create performance issues as the dataset grows in size. But also, your model's performance in terms of quality can degrade over time if you're not constantly evaluating its responses. This can occur because of new info within RAG, a model swap/upgrade, or changes to prompts. Keeping all of those knives in the air is tricky. We're hoping we can solve a bunch of pain points around this so that reliable AI systems are accessible to anyone.

Funny you bring it up. We shipped Vulkan support TODAY through a tight integration Lemonade (https://lemonade-server.ai).

We now support AMD, Intel, CPU, and Cuda/Nvidia.

Hit me up if you want a walk through - this is in dev right now (you have to pull down the repo to run it), but we'll ship it as a part of our next release.

https://docs.llamafarm.dev/docs/models#lemonade-runtime

Likewise, it's been fun building in the open on this one. You can download the CLI with just: curl -fsSL https://raw.githubusercontent.com/llama-farm/llamafarm/main/... | bash

Then do LF init to get the project started!

Yep! We were working on an authentication startup (https://news.ycombinator.com/item?id=30615352) and built it to $1.5M in ARR, but then we saw even a bigger pain point; local AI is hard. When we tried building a corporate knowledge base with RAG and local models, we hit the same wall: a painful gap between prototype and production.

Production-ready enterprise AI requires solving model management, RAG pipelines, model fine-tuning, prompt engineering, failover, cost optimization, and deployment orchestration. You can’t just be good at one or two of these, you have to be great at all of them or your project won't succeed. And so Llamafarm was born!

Monetization-wise - We’re open source and free forever, with revenue coming from enterprise support, managed deployments, and compliance packages—basically, companies pay for confidence, not code.

Oh man, yeah, big pivot.

We still are Rownd (https://rownd.com); but we see the writing on the wall. SaaS Software that helps with "hard code" problems is going the way of the dodo.

What used to take a few weeks and was hard to maintain can be down with Codex in the background. We are still bringing in decent revenue and have no plans to sunset, we are just not investing in it.

We all have IBM backgrounds - not sexy, but we are good at running complex software in customer datacenters and in their clouds. AI is going to have to run locally to extract full value from regulated industries.

We are using a services + support model, likely going vertical (legal, healthcare, and we had some good momentum in the US Gov until 1 October :)).

Hey thanks! Sorry about the broken link - here's a better docs link for now https://docs.llamafarm.dev/docs/intro mind sharing where it's broken?

Just found and fixed a bad link on the bottom of the website - thanks again for pointing that out !

Right...there are lots of ways you could do that. Most of the ways we've seen enabling that sort of thing tend to be programmatic in nature. That's great for some people, but you have to deal with shifting dependencies, sorting out bugs, making sure everything connects properly, etc. Some people will want that for sure, because you do get control over every little piece.

LlamaFarm provides an abstraction over most (eventually all) of those pieces. Something that should work out of the box wherever you deploy it but with various knobs to customize as needed (we're working on an agent to help you with this as well).

In your example (alarm monitoring), I think right now you'd still need to write the agent, but you could use LlamaFarm to deploy an LLM that relied on increasingly accurate examples in RAG and very easily adjust your system prompt.

Good question — that’s actually the sweet spot for LlamaFarm.

You can wire things together yourself (LangChain, bash, Ollama, etc.), but LlamaFarm tries to make that repeatable and portable. It’s declarative orchestration for AI systems — you describe what you want (models, RAG, agents, vector DBs) in YAML, and it runs the same way anywhere: laptop, cloud, or fully air-gapped edge.

So instead of gluing frameworks and breaking them every update, you can do something like:

name: home_guarde runtimes: - detect_motion: {model: "phi-3", provider: "lemonade"} - alert: {model: "gpt-5", fallback: "llama3:8b"} rag: embedder: "nomic-embed-text" database: chromaDB

…and it just runs — same config, same behavior, whether you’re doing local RAG or home monitoring. The goal isn’t to replace the DIY route, just to make it composable and reproducible.

Hey thanks! I'm Rachel from LlamaFarm; we actually use LlamaIndex as one of our components. It's great for RAG, and we didn't want to reinvent what they've already done. LlamaFarm is about bundling the best of open source into a complete, production-ready AI project framework. Think of us like the integration and orchestration layer that makes LlamaIndex, plus model management, plus prompt engineering, plus deployment tools all work together seamlessly.

Where LlamaIndex gives you powerful RAG primitives, we give you the full production system - the model failover when OpenAI is down, the strategy system that adapts from development to production, the deployment configs for Kubernetes. We handle all the boring stuff that turns a RAG prototype into a system that actually runs in production. One YAML config, one CLI command, and you have everything from local development to cloud deployment. :)

The hardest part, honestly, is the runtime. How do we make it super easy actually to deploy this. We are still working on that. Where do you see a few good places to focus at first? I was thinking AWS and Google, since both have good GPU pricing models, but I am probably missing a few good ones!

Great point. I can see how you'd land there. Also a great idea! xD

Maybe a better descriptor is "self-sovereign AI?" "Self-hosted AI?"

Sounds like something that could be implemented with llm-d, though I've not experimented with it.

https://llm-d.ai/blog/intelligent-inference-scheduling-with-...

We're building something closer to this at Muna: https://docs.muna.ai . Check us out and let me know what you think!

Fair question. The core will always stay open source and free. We’ll monetize around it with things like managed hosting, enterprise support, and compliance options (HIPAA, SOC2, etc). Basically, we make money when teams want someone to stand behind it in production, not for using the software itself. But let us know if you have other ideas! We're still new to open source

Our business model is not to compete in open-source; we are just providing this to the community since its the right thing to do and as a signal to enterprises to reduce risk.

Our goal is to target large enterprises with services and support, leveraging deep channel partnerships. We do ship agents as a part of the project and it will be a critical part of the services model in the future.

Appreciate the pep talk, but let’s not pretend infra and developer experience plays can’t scale. GitLab, HashiCorp, and Vercel all "just built better DX for open source" and somehow ended up billion-dollar companies.

Agents will come and go (and probably run into the same orchestration headaches), but someone still has to build the reliable, open foundation they’ll stand on.

more context - https://x.com/swyx/status/1904256213661192405 i really mean it in as non mean spirited a way as possible

This whole comment is predicated on the idea that something is only worthwhile if it will lead to a "venture scale startup" and that to do anything else is to "waste the best years of your life".

I, for one, am glad that not everyone shares your ethos. Conversely, I'm also glad that there are people out there "building agents". Diversity is a good thing. Encouraging everyone to only do one thing is a bad thing.

Oh! Muna looks cool as well! I've just barely glanced at your docs page so far, but I'm definitely going to explore further. One of the biggest issues in the back of our minds is getting models running on a variety of hardware and platforms. Right now, we're just using Ollama with support for Lemonade coming soon. But both of these will likely require some manual setup before deploying LlamaFarm.