Effective context engineering for AI agents (opens in new tab)

These companies all wax on about how important context engineering is yet not one of them has released acceptable tooling for end users to visualize and understand the context window as it grows and shrinks during a session. Best Claude code can do? Warn you when you hit 80% full

The article doesn't really give helpful advice here, but please don't vibe this.

Create evals from previous issues and current tests. Use DSPy on prompts. Create hypotheses for the value of different context packs, and run an eval matrix to see what actually works and what doesn't. Instrument your agents with Otel and stratify failure cases to understand where your agents are breaking.

I find you can give it a task and the full context in your 1st message, and also include (a) asking what files are needed to understand and complete task, and (b) ask if there’s anything ambiguous about the task/question. Then, when you get the response, create a new chat with just the files it recommends, and the ambiguities explained in the 1st comment. Sometimes you need a couple of rounds of this.

The you will have a good starting point, with less chance of running out of space before solving the task.

If you can’t give it full context at the beginning, you can give it a tree listing of the files involved, and maybe a couple of READMEs (if there are any) and ask it see if it can work out what files are needed, giving it a couple of files at a time, at its suggestion.

Why are we hearing that "studies" have "uncovered the concept of context rot as the number of tokens in the context window increases"? It's obvious, and we've always known this.

Agents are stateless, hence the need for context. This means that all they know about the ongoing session is what's in that context (generally speaking). As the context grows any particular element within it becomes a smaller and smaller percentage of the whole. The LLM is not 'losing focus'; it's being diluted with more tokens. But then I suppose anthropomorphism comes naturally to a company named Anthropic, and 'losing focus' does make it sound more human.

They didn't need a study and article, but it likely contributes towards the mystique. Hence the use of phrases like "this results in n² pairwise relationships for n tokens" to make it sound more erudite and revelatory.

I think "output engineering" is equally as important, and steering with grammar (structured output with json schema or CFGs directly) is a huge win there I find:

https://platform.openai.com/docs/guides/function-calling#con...

I’ve been playing around with Apple’s Foundation Models, their on device llm has a 4k context window. That’s really been an interesting exercise in context engineering coming from others like Claude and GPT. I think those larger context windows have made me take context engineering for granted.

It’s kind of useful but I suppose they just admit that failure rate increases with large context windows. My guess is that what happened to the presentation of those Meta glasses where the model would not do what was asked for.

Another interesting thought might be that long horizon tasks need different tooling, and with the shift to long running tasks you can use cheaper models as well. None of the big providers have good tools for that at the moment, so the only thing they can say is: to fix our contexts but still use their models.

I think any meaningful context engineering strategies will be trade secrets.