Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks (opens in new tab)

Even the SOTA models have this problem when the work is complicated enough. The problem is amplified more with the small models.

…but then all of your metrics and data seem to focus 100% on accuracy. You need to address speed."

I wonder, if you were to use cloud-based LLMs more often, you might find that accuracy (fidelity?) is indeed more more lacking in your local models.

You can always just throw hardware at your speed problems after all.

On a per-call basis, the wrappers are pure python ifs and such, measured in ms easily, and frankly negligible compared to the LLM call itself which will be on the order of magnitude seconds.

Where timing gets interesting is that forge will slow down workflows because the retries mean you don't error right away. Bare runs were failing fast in my experience. But on a per-call basis there's very little overhead.

I haven't detailed it simply because the order of magnitude of a single LLM call is so much higher than all the overhead put together.

A few things I noticed related to your points: - on conversation rewind, I implemented a similar tool call collapse on the main agent (the one you chat with). Once it was done with a task, the tool call history was collapsed to keep the context clean - it was more about hygiene than size.

- the harness interrogating the model bit is a bit different, I haven't tried that approach. Forge relies on model self-correction in a bid to avoid having bespoke error modes, but I guess if you can abstract and automate the interrogation based on schema or something that could work!

Overall I like the clean conversation history aspect, but I suspect that you might be doing a lot of round trips for tools with many args, versus "letting it fail and giving it one nudge". That being said, it's an interesting idea for harder scenarios/tasks!

Unfortunately I am caught up right now in other projects at work and otherwise and just tried few dozens of prompts to see if this is even achievable.

However, that doesn't explain the Lamaserver prompt vs llamafile at ~ +4pts, or vs Ollama (at ~ +30ish pts) that sits almost perfectly between llamaserver native and llamafile.

The backend affects almost all model families, and was just something I've never seen really talked about.

I think this sits a level higher than Forge - maybe testing the workflow proper and integration points that it might surface (if some tools are giving access to an MCP or something).

Could likely layer both together without much trouble.

Only thing I'd be curious about is how you handle the non-deterministic nature of these models. Sometimes they get the tool call right, sometimes they barf bad json. Does the suite run multiple trials?

That would certainly work in theory, but I'm not as familiar with parallel calls.

- If you mean the model calls the tool twice, identically, in a batch call - that would work fine and Forge handles batch calls, but many small models wouldn't think to do that so you'd have to explicitly prompt it to do so.

- If you mean ask the LLM twice to call the tool and look at both answers, my only concern would be latency from doing 2 calls instead of 1.

- Unless you're truly running 2 instances of the model and aren't memory-bandwidth bound, then yes running parallel workflows would likely help. Especially if you could have them compare notes at certain steps or something.

But I haven't explored this much at all so if you're thinking of something else, let me know!

I gave it 3 simple changes to make. It did it perfectly.

Then I tried with a much smaller model. It also did it perfectly, except 3x faster and 9x cheaper.

I used to think "best model" was what's at the top of the benchmarks, but for most tasks that just means you're going to wait longer and pay more money. The right model depends on the job.

(Also, speed itself is a feature -- when you get the really fast models, it enables a kind of real-time interactive usage that is otherwise not possible in the "alt tab and hope it's done" workflow.)

The biggest challenge has been balancing the desire to hyper optimize for my favorite models, versus average behavior, versus consumer needs.

I'm in the same boat, tuning models wasn't super interesting, though I might do a focused spike on behavior -focused fine tuning. But the harness matters almost more than the model in many cases.

Big frontier models need this less than small models.

Forge doesn't have a SWE-specific eval, but I've built a custom coding harness (not public yet but maybe soon) built on forge and saw the same behavior you seem to have seen in agentic coding.

I'd assume they could be combined. A coding harness would own the agentic workflow by nature, forge guardrails would help tool calling.

I haven't given it a thorough read yet but I think their guardrails might be more focused on the workflow level. They are doing error capture at tool level with warnings to the model, but I'd need to dig deeper. On the surface definitely the same design philosophy! Maybe Forge makes error nudges more of a first-class citizen?

Our compaction strategies might be the most similar of all the pieces. Cool find!

You don't have to define the workflow steps. You can just expose the set of tools to the model and let the LLM call whatever it wants in any order, and every guardrail except the prerequisite step enforcement is still there to help.

If your workflow does have step enforcement, that can also be conditional. For example like Claude code does read required before edit. You can define a conditional enforcement where the agent must have called read before edit, and even force the same file path. That doesn't mean the model has to call edit at all...

But maybe I could have been clearer in the docs on the workflow pieces.

Forge sits one level lower - in my mind - than a gate which would sit more at the workflow level. Perfectly complementary.

Forge is just trying to make sure that when the model decides to do something, thee execution is reliable.

As for software integration, let me know if you run into any issues and I'll be happy to take a look or try to patch something!

Harnesses as first class infra all the way. I'll take a look at your work and see if I spot any obvious tensions.

Name was just a portmanteau of Calcifer's forge, because Howl’s moving castle seemed like a good metaphor for what I was trying to do… I had synthetic models as apiece there but I realized a) it was out of place and b) it was my favorite feature there

The proxy mode should integrate seamlessly, and the middleware guardrail mode could be lifted into pi.

As for little coder, I love it! I wanted forge to be more generic than just agentic coding as there's many more agentic workflows worth optimizing with small models.

The isssue/use-case is more around, say, a database table or legacy systems where your tool is just hitting a legacy API that may or may not be good. A surface you don't control.

It didn't come up as a use-case in this eval honestly, it's more the concept of a standard, like 4xx vs 5xx. I just felt it was missing from the ecosystem overall.

Harnesses do have retry mechanisms. In opencode in particular, I think they return the error as-is to the model in the next turn. But that's slightly different. Harness retries come mostly in two flavors:

1) provider-layer: HTTP requests to cloud retries, with or without exponential backoff. It covers you for transient network hiccups or rate limits, and a big Opus model really doesn't need more than that.

2) sort of a hope-and-pray retry. Tool ran, returned an error string of some kind, gets fed into model as-is, and the model is expected to read the error message and self-correct with no guidance. This is fine for frontier, and even some of the large oss models. They have the context-following capabilities needed. For smaller models, this won't be enough, not reliably over many turns.

- if model outputs malformed json, provider will reject it before it even reaches the tool, the error loop is broken. A rescue parser handles that - can be ~5-15% of calls on a small model sometimes.

- model calls the wrong tool, correctly, then proceeds confidently with context that won't help it. step enforcement can help here.

- model terminates prematurely, thinking it's done. prerequisite enforcement can help here (say, forcing the model to call pytest before declaring the feature built).

- Escalating nudge messages, that specifically nudge. Just returning error messages doesn't tell the model what to do, it just tells it it was wrong. A message that spells out "tool X does not exist, call one of the available tools: A, B, C" is more helpful to a small model than "error: X not found".

So, in short - yes, retries exist in harnesses, but rely on top-tier model interpretation of the error messages. When working with top models, there's likely no real difference, or a minor one (see Opus bare vs Opus reforged). But Forge provides a more hardened suite of guardrails that are effectively necessary for small models.

What project are you referring to specifically?

Some of this is inside the model, like topic refusals. Forge sits at the tool call level.

My personal workflow uses guardrails at the SDLC level: I have a standard pipeline (plan, design, code, build, test). I use gates between each stage, and the right composition leads to a much higher quality in the final product.

Also worth mentioning that gate failures are given to the agent that produced the artifact, so it has a chance to fix it. That means that I don't have to review obviously wrong output.

You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.

The key I think is to look at what use cases you have that aren't big monsters. Auditing logs, home assistant, reading and summarizing news rss feeds, etc...stuff that's fairly bite-sized per task, but high volume. Then the local models make sense and they just need mechanical reliability to close the gap.

/v1/chat/completions is the entry point.

In proxy mode, here's what forge applies on each request (handler.py builds these):

Response validation: ResponseValidator(tool_names) checks each tool call against the declared tools array. If the model emits a call to a name not in tools[], or a malformed call shape, it's caught before the response goes back.

Rescue parsing: When the model emits tool calls in the wrong format — JSON in a code fence, [TOOL_CALLS]name{args} (Mistral), <tool_call>...</tool_call> (Qwen XML) — rescue parsers extract the structured call and re-emit it in the canonical OpenAI tool_calls schema. This is the biggest practical lift, especially on Mistral-family models that ignore native FC and emit their own bracket syntax.

Retry loop with error tracking: ErrorTracker(max_retries=N) — if validation fails, forge retries inference up to N times with a corrective tool-result message on the canonical channel, rather than returning a malformed response to your caller. From your perspective the proxy looks like a single request that just took a few extra ms.

What proxy mode does NOT do (because it's single-shot, not multi-turn): prerequisite/step enforcement (those need a workflow definition spanning turns), context compaction, session memory. For that surface you wrap the WorkflowRunner class in Python — proxy mode trades that depth for "use forge with your existing setup, no Python rewrite."

So yes — the proxy is fortifying the response shape and retry behavior of /v1/chat/completions. The full agentic guardrails are at the Python class level above it.

For greenfield projects, I've been building on forge native using WorkflowRunner so I get all guardrails. But obviously as a drop-in replacement in existing systems then proxy is the way to go.

Proxy mode should work fine with remote models, the only constraint is the compatible endpoint - which is standard anyways. I don't think you'd have any issue hitting either a remote gateway like liteLLM or just claude API.

Latency is dependent on the guardrails firing, effectively. If nothing fires, it's a passthrough, for all intents and purposes, very little overhead. But if a retry nudge fires then that's another LLM call.

As a consumer for a home assistant, a retry nudge firing is something I'd catch, and have my voice model output a pre-baked "one sec, trying again" sort of filler message or something.

I run small models at home, so I'm very curious.

At least, if I understand your economic benefit angle correctly.

For scenarios to get inspired by I'd look at those tagged "model_quality" or "advanced_reasoning".

I think we share a lot on tool definitions/schemas. Forge will let a consumer define a tool, set of tools, pydantic schema for each, etc. outlines seems to be similar with their task definition.

I think where we differ is what happens when that doesn't work...and the model still doesn't get the contract right. Something like a pydantic-valid string path for glob, that points to a non-existent thing. Glob will error, forge catches, and nudges the model. Forge does very little model output manipulation (just a basic regex parse to try to find json/XML), the core of it is in the retry mechanisms.

Once I dig into it more I'll try to highlight other deltas.

Plus it's cool to see a little 8B model writing code :)

A lot of current tooling is layered mostly at the workflow level. Auth for the agent, or memory management for the agent (like some smart skills stuff), but Forge sits below that.

In most cases I've looked at, it could be slotted in with other work without much disruption. Forge just increases mechanical reliability of tool-calling, it shouldn't disrupt your workflow-level layers much.

But at the end of the day, if the model keeps responding with text, there's nothing forge can do. I've run into that failure mode for sure, even with forge.

That works well enough for all the models shown in the eval here: relatively modern 8B+ models.

But some of the older generation (mistral 7b, that sort of thing) still can't be reliably used in something like a production setting.

I just need more GPU wall clock time to get more evals done. ETA is...a few weeks? Got distracted by the coding harness.

But the results are the same. Reforged models do better than bare, even at those sizes. As for published results, I ran forge on Anthropic models and reforged doe better than bare for them as well :)

Context compaction can also affect the outcome - I have eval scenarios for that as well but not in the published set, only in the repo. For those, I'd say "it's better than nothing". If you hit max context, the whole thing will barf or OOM the rig or something like that. So compaction degrades performance versus some theoretical ideal where you never need to, certainly. But it's better than a hard failure. Eval on those scenarios showed increasing degradation depending on severity of compaction. I view the auto-compaction as insurance. I never give the models tasks that will require that much context, but if it ends up getting there then the run might be saved.

Scenarios range from basic 2-step workflows, to more complex ones with dead ends, breadcrumbs, misleading names.

Concrete example: Task: get, analyze and report on Q3 sales data.

Model emits: analyze_sales(quarter="Q3"). This skipped the fetch step. Forge's response validator catches it before the tool function runs. Instead of letting the bad call hit the real impl (which would error or hallucinate), forge replies on the canonical tool-result channel.

We send this to the model: tool_result: [PrereqError] analyze_sales requires fetch_sales_data to be called first. Available next steps: fetch_sales_data

Model emits a corrected fetch_sales_data(...) on the next turn.

Three enforcement paths use this same channel: prerequisite violations, premature terminal calls, unknown-tool retries.

We also have rescue parsing for known templates (Jason OpenAI style, XML like granite, etc) where we try to parse tool calls that might be malformed.

And lastly bare text response nudges. Small models love to chat, we need them to call tools!

I do think there's some differences though. The biggest one being that forge isn't a coding harness, it's a guardrail primitive, really. Applicable to any tool-calling workflow.

As for the errors, are you nudging or passing errors back or swallowing them completely? Love the 2-stage routing though, neat!

But otherwise, forge really doesn't own or opine much of the workflow. Step enforcement exists if you want it, so do prerequisites, but the idea is that those could be conditional or optional (you may never need to edit a file).

The guardrails are designed to work for non deterministic flows or deterministic ones. In the latter, you just might not have one of the guardrails active. It's much more about nudging the model back on track than laying more obvious tracks, in a sense.

Overall, agentic reliability is definitely an active field.

Always happy to see folks looking into small local models!

Dashboard is in here: https://github.com/antoinezambelli/forge/tree/main/docs/resu...

Without forge, I'd guess a small model used for Hermes would have to retry entire workflows when an uncaught exception triggerd when it tried to reply with text when "calling a tool" ("Here is the tool call: [json blob]"). The issue there becomes partial successes can lead to state changes that need to be addressed (it booked the flight already, home it doesn't double-book).

Forge won't help with model reasoning quality though. If it the model thinks the right thing to do is to book 3 buses for your trip, forge doesn't care, it'll just make sure those api calls land.

And if you didn't mean that then please elaborate :)

I did go with an extreme example in the post (but true). Other deltas are smaller but still statistically significant. 30 pt swing between llamserver prompt vs ollama, 4-5pt swing between llamafile and llamaserver prompt.

ToolResolutionError is really inspired by HTTP 4xx vs 5xx codes. I don't even have a super clean abstraction I'm happy with yet, I just noticed a lack of standard in the industry (that I was aware of) so I thought to surface it as a gap. I'm sure there's a better shape than my current ToolResolutionError but it's a start!

- First, there's totally a "risk" there. I built both the harnesses and the eval suite and that's hardly a double-blind study. There's no world where some bias doesn't leak through.

- I did try to design the guardrails to be domain-agnostic so they aren't tuned to specific scenario failures and return generic nudges to the LLM.

- Most tactically, the guardrails were built on the first 18 scenarios (OG-18) published in the paper, and only after did I had 8 more advanced reasoning ones. I didn't update the guardrails when I added those, and the lift was still there. If they were overtuned, they wouldn't have the same level of impact on an newer set.

- I did dogfood forge post publication using several unrelated consumers and the features I baked in were rarely guardrail related. If they were, it was more model focused (ie, xml-parse-rescue for granite models).

But at the end of the day, there's an explicit connection between the guardrail author and eval author. Happy to take contributions of eval scenarios if you want to stress test things, or hear about your experience running a completely different consumer!

Maybe I've been spending too much time reading the evals and I now sound like an LLM...

Either way, here I am - happy to answer any questions!

The original post and every comment by OP is so full of AI slop ("the biggest surprise!", "one thing I didn't expect!", "the biggest challenge!", etc. etc.") that is absolutely painful to read. I still can't believe most people (especially here on HN, I thought we were a bit better than this) can't notice all this stuff.

What's much worse, it's that all these people posting this useless slop are so dishonest ("I definitely use LLMs to help write things - but this is my draft!") that it makes me really nauseous... This is the worst time to be an internet user if you have more than 2 points of IQ.