Same Model, Different Floor

Same tasks, two harnesses: the frontier model paid $2.13 a task raw vs $0.59 governed; the cheapest model went 31% to 94%. The difference was never intelligence — it was the floor under it.

This month a vendor published a benchmark that is more interesting than its leaderboard.

DevRev’s Enterprise-Bench evaluates AI agents on cross-system enterprise questions (which customers does this bug affect, what’s blocking this deal) against a synthetic mid-market company with years of accumulated noise: 32,768 support tickets, 8,448 engineering issues, 8,704 opportunities, and the punchline that only about 0.16% of it is relevant to any given question. The published results compare their agent against Claude Code. Their agent wins. A vendor’s agent winning the vendor’s benchmark is Tuesday.

Here’s what isn’t: both lanes run the same model. Claude Opus underneath their agent; Claude Opus underneath Claude Code. When the model is held constant, a leaderboard stops measuring intelligence and starts measuring harnesses — and their methodology says so in plain text. Moving an agent from curated tools to protocol-realistic APIs costs 18 to 19 points of accuracy and doubles token cost, by their measurement. Their conclusion: retrieval architecture predicts production performance better than model choice.

A vendor claiming its architecture is the special ingredient is marketing. But this claim is checkable, because they published the dataset under Apache 2.0. So I checked it.

The experiment

The benchmark ships protocol-realistic mock servers — a Salesforce facade with real SOQL and faithful error messages, a Jira facade, a Drive facade. That’s the “raw” arm: the systems as an agent actually meets them.

I added a second interface over the same data: a thin governed tier that does what my context layer does in production. Joins computed at the data layer. Exclusions applied — and reported, not silent. Policy thresholds cited to the binding documents they come from. And on every response, a fingerprint: which records and which documents this answer actually consumed.

Then the same tasks, the same LLM judge, the same models, through both doors — Claude Opus at the frontier end, matching the benchmark’s own setup, and Claude Haiku at the cheap end. First as single exploratory trials, then as a 75-trial grid: eight trials per cell for Haiku, three for Opus.

The frontier model passed everything on every interface — twelve trials of twelve — but on raw surfaces it paid for the privilege: up to 29-turn trajectories averaging $2.13 a task, against $0.59 for the same model on the governed tier. For a frontier model, the floor is pure economics: same accuracy, a third the cost, a fifth the turns.

For Haiku, the floor is the difference between working and not. On raw surfaces it passed 5 of 16 trials — its signature failure was silently dropping records it never knew existed, the exact failure the noise design exists to induce. On the best interface for each task, the same model passed 15 of 16. Thirty-one percent to ninety-four percent, no model change. The best cells are stark: on the aggregation task, where every thin interface scored one to three passes out of eight, the governed tier went eight for eight at four cents a run. Fixing the weak model by upgrading the model costs about $1.70 a task, forever. Fixing it by upgrading the interface costs a dime, once the tier exists. That ratio is the vendor’s headline claim, reproduced independently, on open substrate, by someone with no stake in their product.

Where the money actually goes

The mechanism is visible in the turn counts. Thirty-three turns of a frontier model working raw surfaces is not thirty-three turns of judgment. It’s join reconstruction: which ticket belongs to which account, which component rolls up to which product area, which document holds the thresholds — rediscovered from scratch, on every task, forever. The model’s capability is being spent as a shovel.

Cerebras published the same discovery from the inside this month: their internal knowledge base handles 15,000 questions a day, and every load-bearing mechanism in it is an exclusion mechanism — thresholds before a message earns an embedding, stale answers demoted, summaries embedded instead of transcripts. Different company, different layer, same law. The systems that work aren’t the ones that retrieve the most. They’re the ones that refuse the best.

The confession

Halfway through, the experiment turned around and audited me.

My governed tier shipped with a wrong number. I had misread an ambiguous cell in the benchmark’s service-level agreement — the operative threshold sits next to a parenthetical baseline, and I took the parenthetical. The judge’s rubric carried my same misreading, so the governed arm “passed” while wrong. The raw-surface agent, reading the binding document directly, applied the correct value, and my rubric failed it for being right.

A curated layer whose values aren’t verifiably derived from the sources they cite isn’t governance. It’s a new hallucination surface with better formatting. The fingerprint on my tier said “sourced from the MSA, binding authority” while the constant contradicted the MSA. The fix wasn’t just correcting the number; it was the rule the incident taught: every compiled value carries a derivation check against its cited source, on a re-verification horizon. Curation needs the governance it provides.

The two-hundred-line floor

Then I asked which part of the governed tier was doing the work — the pre-computation, or something smaller.

So I built a stripped arm: raw surfaces plus one document. A linkage map — about two hundred lines of declarative topology. Which entities exist and where. Which fields join which. That tickets relate to engineering issues only through shared components, never through text similarity. That thresholds live in the MSA documents and nowhere else. That statuses are lowercase, and that a record created after your reference date has negative age and must be handled out loud. No computed values. No answers. Just the map.

The weak model went from fail to pass.

That’s the part I’d send to anyone building in this space: the model didn’t need the data pre-joined. It needed to know how the sources relate — the topology the systems themselves never tell you. Platform vendors solve this by asking you to migrate everything into their graph, where the map is theirs and compounds in their account. The map doesn’t require the migration. It’s a description, not a destination.

One more finding I wasn’t looking for. In an early configuration, the stripped arm could see the governed tools but wasn’t permitted to use them — and the weak model stalled at the locked door instead of using the open ones. Hiding the denied tools fixed it. The tool manifest, it turns out, is a policy surface: advertising a capability you won’t permit degrades an agent more than its absence would.

Floors have rungs

Put the pieces in a row and a ladder appears. At the bottom, the agent improvises from the map — general, honest, expensive. One rung up, it follows a written procedure for a task family: cheaper, sharper, and only as good as the agent’s ability to execute a recipe. At the top, the procedure is compiled into the floor itself: the server executes, the agent inherits, and the run can’t flub a step it never takes.

Nothing about that ladder is new to readers of this series. It’s the same grammar as autonomy: a task family climbs by evidence: earned, recorded, reversible. And the map at the bottom has its own life cycle. The derivable part (entities, keys, observable conventions) is a scanner anyone can run on day one. The part that matters most can’t be scanned: the watchouts, the reading rules, the never-do-this entries. Every one of those in my map was born from a specific failure, including two of mine. That knowledge is scar tissue. It accrues only to whoever is operating, and it compounds in whoever’s account the operating happens.

What this means for the bill

The industry prices intelligence by the token and argues about models. These measurements (small, early, and open to anyone who wants to rerun them) say the argument is misplaced. The floor converted a frontier-model workload into a cheapest-model workload without losing a single required criterion. Model spend is an operating cost, paid on every task, forever. The floor is architecture: built once, inherited by every operation that stands on it, and improved by its own failures in a way per-token spend never is.

The last essay argued you shouldn’t commission the governance floor — you should inherit it. This one is what happened when I priced the difference. Same model, different floor. The floor is the product.