A Memory of Judgment

I tore down the five best open-source memory systems. Every one remembers what's true. None remembers whether the agent should be trusted to act — and why nobody built that is the whole business.

I spent last week doing something unglamorous. I cloned the five best memory systems in open source — the ones an AI agent uses to remember things across sessions — and read them line by line. supermemory. mem0. Open Brain. Graphiti. Letta, the project that used to be called MemGPT. Between them they have well over a hundred thousand GitHub stars and a good deal of venture funding. These are the serious ones.

I went in looking for the thing my own company bets on, to find out if I was about to be commoditized the way the agent harness just was. I came out with a cleaner finding than I expected, and it reframes the whole category.

There are two kinds of memory. The entire industry is building one of them.

The kind everybody builds

The first kind is memory of what is true. Your name, your preferences, the fact that this account renewed in Q2, that this vendor’s invoices put the PO number in the footer. Call it knowledge memory. It’s what “give the agent a memory” means to almost everyone, and the five systems I read are genuinely, impressively good at it.

They’ve solved real problems. mem0 has an open reconciliation engine that notices when a new fact contradicts an old one and updates instead of just piling on. Graphiti is bi-temporal — it tracks not only what’s true but when it was true, so you can ask “what did we believe about this customer in March” and get a real answer. Letta gives the agent editable memory blocks it rewrites mid-conversation, an idea straight out of the original MemGPT paper. Open Brain has a genuinely thoughtful provenance model. This is good engineering by good teams. I’m not here to run it down.

I’m here to point at the thing all five of them are missing — the same thing, in exactly the same place.

The kind nobody builds

The second kind is memory of judgment. Not what’s true about the world — whether the agent’s own past actions were any good, and therefore how much it should be trusted to act again without someone checking.

Every one of these systems remembers facts. Not one of them remembers whether its agent has earned the right to act. Five for five.

And the tell is almost too perfect. Letta is the most sophisticated of the five, and its pitch is literally “stateful agents that learn and self-improve over time.” So I went looking for the learning. There’s a field in the code called feedback — you can mark a step thumbs-up or thumbs-down. Promising. I traced every place that field is read.

Nothing reads it. It’s a column. It goes to a dashboard a human might look at. No part of the agent consumes it, nothing counts the thumbs-up against the thumbs-down, nothing changes what the agent is allowed to do next time. The single most advanced open memory system has a feedback signal wired to a light bulb. The loop is drawn on the box and not connected inside it.

That is the whole industry in one detail. Everyone can capture what happened. Nobody closes the loop from what happened back to what the agent is trusted to do.

Why the loop is worth closing

Here is what closing it looks like, and it’s not exotic.

Every time a human approves an AI action untouched, that’s a verdict: good judgment, this class of work. Every time a human edits it before approving, the edit is a labeled correction — the richest signal an operation produces. Every reject is a boundary. Accumulate that stream per type of task and per client, and the agent’s autonomy stops being a slider an admin drags and becomes a function of its proven track record. It earns the right to act unsupervised on the reversible, low-stakes work it has demonstrably gotten right — a hundred clean approvals in a row — while staying gated on the consequential work, and demoting itself automatically the first time the evidence turns. Trust that is earned, per class, from evidence, and revoked the moment the evidence sours.

That’s a memory too. It’s just a memory of the agent’s own judgment instead of the world’s facts. And it’s the one that compounds into something a competitor can’t clone, because what accrues isn’t an algorithm — it’s two years of your team’s corrections on your operation, and the standing to act on them.

The honest objection — and why it’s the whole point

You could say: that’s just knowledge memory with a feedback signal bolted on. Capture the thumbs-up, feed it back, done. Letta already has the field.

Letta is the proof that it isn’t that easy — and why it isn’t is the entire business. The hard part was never capturing the signal. It’s two things a memory library sold to developers structurally cannot have.

The first is a human generating real verdicts every day. A developer who pip-installs a memory SDK is not sitting there approving, editing, and rejecting the agent’s work on a live operation. There is no judgment stream to learn from, because nobody is in the loop making judgments. You only get that stream if someone is operating — running the actual work, catching the actual mistakes. The signal is a byproduct of operating, and a tool vendor doesn’t operate.

The second is the nerve to let the agent’s permissions actually move based on it. A system that quietly grants itself more power is precisely what no regulated buyer will ever sign. Making earned autonomy safe instead of alarming means every graduation is itself an entry in a record you can prove wasn’t altered after the fact — this class of work earned supervised-auto on this evidence, on this date, here’s the one reject that demoted it. That’s a governance discipline, not a feature.

So the gap in all five systems isn’t an oversight the next release fixes. It’s a fork in the road. Knowledge memory is the thing you can build and sell to a developer. Judgment memory is the thing you can only build if you run the operation and are willing to own the governance of an agent that gets more autonomous. One is a library. The other is a company. That’s why the best-funded memory teams in the world all stopped at the same line — not because they missed it, but because the far side of it isn’t a memory product at all.

The buyer’s move

If you’re evaluating anyone who says their agent “has memory” or “learns,” the diagnostic is one question, and it separates the two kinds instantly:

Your system remembers what’s true about my business. Does it remember whether your agent has earned the right to act without me checking — and can you show me the ledger?

A vendor selling knowledge memory will answer about recall: it remembers your preferences, your history, your documents. Good — table stakes. Then watch them get vague on the second half, because “it learns” turns out to mean “it remembers more facts,” and the agent needs exactly as much babysitting in month twelve as it did in month one.

A vendor built for the far side of the fork will answer with a mechanism and show you the record: this task class is at supervised-auto for this client, here’s the accept/edit/reject history that earned it, here’s the entry where it demoted itself last month and why. That’s not a better memory. It’s a memory of judgment — and it only exists if someone is doing the judging.

The bias, declared

I run a company built on the far side of that fork, so discount accordingly. We treat knowledge memory the way What Compounds argued you should treat the harness — as commodity substrate you adopt, not a moat you defend. I tore down these five systems precisely to take the best of what they do — mem0’s reconciler, Graphiti’s sense of when a fact was true, Open Brain’s provenance — and stand it up on our own stack. That part is a build, and I’m happy to have five excellent open references for it.

The part I can’t download from anyone is the memory of judgment, because nobody is selling it, because you can’t build it unless you operate. That’s not a gap in their products. It’s the shape of the whole market: the moment memory stops being what is true and becomes what has been earned, you’ve left the world of tools and entered the world of running someone’s operation on their regulated data, every day, and being accountable for it. That’s not a worse business than selling a memory SDK. It’s a different one — slower to start, and much harder to take away once it’s yours.

Everyone is racing to make agents remember more. The question that will actually sort the winners is quieter: does your agent remember whether it should be trusted — and would you let a regulator read the record of how it decided?