I kept hearing the same pattern: some teams are shipping 10-15 AI PRs daily without issues. Others tried once, broke production, and gave up entirely.
The difference wasn't what I expected– it wasn't about model choice or prompt engineering.
---
One team shipped an AI-generated PR that took down their checkout flow.
Their tests and CI passed, but AI had "optimized" their payment processing by changing `queueAnalyticsEvent()` to `analytics.track()`. The analytics service has a 2-second timeout so when it's slow, payment processing times out.
In prod, under real load, 95th percentile latency went from 200ms to 8 seconds. Ended up with 3hh of downtime and $50k in lost revenue.
Everyone on that team knew you queue analytics events asynchronously, but that wasn't documented anywhere. It's just something they learned when analytics had an outage years ago.
*The pattern*
Traditional CI/CD catches syntax errors, type mismatches, test failures.
The problem is that AIs don't make these mistakes. (Or at least, tests and lints catch them before they get committed). The problem with AI is that it generates syntactically perfect code that violates your system's unwritten rules.
*The institutional knowledge problem*
Every codebase has landmines that live in engineers' heads, accumulated through incidents.
AIs can't know these, so they fall into the traps. It's then on the code reviewer to spot them.
*What the successful teams do differently*
They write constraints in plain English. Then AI enforces them semantically on every PR. Eg. "All routes in /billing/* must pass requireAuth and include orgId claim"
AI reads your code, understands the call graph, and blocks merges that violate the rules.
*The bottleneck*
When you're shipping 10x more code, validation becomes the constraint; not generation speed.
The teams shipping AI at scale aren't waiting for better models. They're using AI to validate AI-generated code against their institutional knowledge.
The gap between "AI that generates code" and "AI you can trust in production" isn't about model capabilities, it's about bridging the institutional knowledge gap.
