Agent memory vs RAG: the diary, not the library

The most common mistake in agent design right now is assuming RAG and memory are the same tool. They answer different questions, and conflating them is why so many “memory” features feel like a worse search box.

RAG is a library

You have a static corpus (docs, tickets, a codebase), and at query time you retrieve the passages most relevant to the question. The corpus does not change because the agent read it. RAG is lookup over what someone already wrote down.

Memory is a diary

Memory is what the agent itself learns over time (the user moved to Berlin, this customer prefers email, that deploy failed for reason X), accumulated across sessions, reconciled when it changes, and recalled later as the agent’s own belief about the world. Memory is write as much as read.

That write is the whole difference. A library does not get poisoned by being read; a diary does, because the diary records what the reader concluded, and a reader can be manipulated.

Three things memory needs that RAG does not

Reconciliation. Facts change. “Lives in Berlin” becomes “lives in London.” A diary must update the fact while keeping the old version in history, not just pile up contradictions. RAG never faces this: the corpus is fixed.

What was true at a given past date. “What did we believe on 5 March?” is a memory question. A library has no belief state to reconstruct.

A trustworthy write path. This is the one nobody talks about. If memory records what the agent learned, then who checks what gets learned? An agent that reads a poisoned web page can write the poison into its own diary as a trusted fact, and every future decision inherits it. RAG’s corpus is curated by people up front; a self-building diary is curated by the same agent that can be manipulated.

The question that reframes the category

Most “agent memory” products are really auto-appended diaries: the working agent decides what to write and writes it directly. That is convenient, and it is also the injection surface. The interesting question for the next generation of memory is not “can it retrieve?” Vector search solved that. It is “can you trust what got written, and can you prove who wrote it?”

That is the axis we build Gather on: memory where a separate reviewer approves every write (the agent proposes; an independent stage commits), so the diary records what is checked, not just what is read. If you want the receipts, the write path shows the same planted instruction a normal memory saves every time, and ours rejects every time.

RAG gives your agent a library. Memory gives it a diary. Just make sure something trustworthy holds the pen.