“The agent forgot” is an outcome label. It does not tell you whether the information was available at the source, saved correctly, retained at the relevant revision, retrieved, placed into working context, causally used by the model, or reflected in the final action. Treating every incident as a retrieval problem risks repairing the wrong component.
Debug memory as an information path: source, write, retained representation, retrieval, context placement, utilization, and effect.
This practice turns the broad agent-memory failure modes into an evidence-backed diagnosis. Agent trajectory observability supplies the execution artifacts needed to trace that path.
The seven-stage path
A memory diagnosis begins with explicit contracts between stages. Each stage consumes an artifact and produces another artifact that can be inspected.
| Stage | Meaning | Evidence to inspect |
|---|---|---|
| Source | The interaction, document, or event from which the information should originate | Exact source text, sender, time, and scope |
| Write | The extraction, update, consolidation, or deletion operation applied to the source | Operation input, instruction, output, and status |
| Retained representation | The form that persists after the write | Saved bytes or record, path or key, schema, version, and revision |
| Retrieval | The process that selects retained information for the current task | Query, candidates, scores, filters, and selected results |
| Context placement | The assembly of selected information into the model’s active input | Exact prompt or context snapshot, ordering, truncation, and conflicts |
| Utilization | Whether placed information causally affects generation | Context-answer consistency, controlled substitutions, and repeated replays where possible; no single answer or reasoning trace proves causal use |
| Effect | The externally observable result | Delivered response, tool call, state mutation, or downstream receipt |
Use symptom words only as starting hypotheses. Forgetting may involve retrieval or placement, distortion may enter during writing or a later update, and a correct generation may still become an incorrect external effect. The artifacts determine which stages remain implicated.
The MemTrace preprint studies memory construction, update, retrieval, and answer generation through execution graphs. MemTrace defines its target as the earliest decisive faulty operation: an operation whose predecessors are functionally correct and whose corrected output would rescue the result under ideal later execution.
Freeze the incident before changing it
Create a reproducible incident record before teaching the agent the missing fact again. Capture the triggering request, actual result, expected result, source evidence supporting that expectation, relevant conversation or run identifiers, and the observed effect. Preserve the original state until those artifacts have been captured; reteaching first prevents a clean comparison with the failed run.
Ground truth needs its own audit. The MemTrace authors report benchmark annotation errors caused by imprecise questions, insufficient source evidence, and inconsistencies between reference answers and supporting evidence. Prove that the expected answer follows from the available source before assigning a memory failure. If the source does not support the expectation, the incident belongs to task specification or evaluation rather than memory attribution.
Trace backward, identify divergence, and keep looking
Begin at the observed effect and walk backward through concrete artifacts:
- Verify what was actually delivered or executed, rather than stopping at what the model generated.
- Inspect the generated answer for agreement or conflict with the assembled context. The answer establishes consistency or conflict with the assembled context; by itself, it does not establish causal use.
- Inspect the exact assembled context and determine whether the relevant representation was present, legible, correctly ordered, and free from unresolved conflicts or truncation.
- Inspect retrieval inputs and outputs, including the query, filters, candidates, scores, and selected items.
- Read the retained representation at the exact revision and projection used by the failed run.
- Inspect the operation that wrote or later modified that representation.
- Compare the write operation’s input with the original source.
Mark the earliest observed divergence, then continue inspecting downstream stages. An upstream fault does not establish that later operations worked correctly. Retrieval can be cleared only when its recorded inputs and outputs show that it faithfully selected the available representation according to its contract. If a write produced the wrong representation, retrieval may faithfully select that wrong representation, fail independently, or do both.
Do not force every incident into a single-cause label. The paper also notes that multiple independent faulty operations may jointly cause a failure, making singleton attribution insufficient. One incident may therefore retain several implicated operations until the trace or an intervention rules them out.
MemTrace’s idealized correction is part of its problem definition, not a replay performed in practice. A practical reading of MemTrace’s counterfactual definition is to use controlled substitutions and repeated replays when the system permits them. Replace one intermediate artifact, such as a retained representation, retrieval result, or assembled context, while holding the rest of the test as stable as possible. Observe whether the failure persists, and repeat the comparison when generation is stochastic.
For utilization, demonstrably present context combined with a conflicting answer makes utilization a leading hypothesis rather than a finding. Compare runs in which the relevant context is held fixed, removed, corrected, or replaced. Those interventions can strengthen or weaken the hypothesis, but neither output agreement nor a reasoning trace alone proves that the model causally used or ignored the information.
Read the storage architecture literally
The MemFS documentation describes MemFS as an agent-owned Git repository projected onto the current computer as an ordinary checkout. Files under system/ are loaded into the system prompt on every turn. Files elsewhere remain outside active context until needed, while the file tree remains visible in the system prompt as a set of retrieval signposts.
A diagnostic reading of that architecture is narrower: a correct file at the exact revision and in the exact projection used by the failed run establishes a retained representation at that stage. It proves nothing about retrieval or context placement. Under the same reading, a stale file under system/ at that revision is a placement hazard even when task-specific retrieval selects the intended material.
Revision state matters. For cloud-backed agents, edits remain local until committed and pushed; another projection can then receive the change. Compare the revision and checkout used by the failed process with the revision containing the expected memory. A correct edit in another checkout or at a later revision is not evidence that the failed run retained it.
MemFS has no semantic or vector index by default, and conversation-history search is separate from file memory. Optional keyword, semantic, or hybrid search requires separate tooling and configuration. Verify which search mechanism actually ran instead of inferring one from the presence of a memory file.
The memory documentation describes additional write paths. Dreaming uses background subagents to review recent conversations, consolidate lessons, and update memory. When a fact was initially correct and later changed, include those background updates in the write history. The same documentation describes /doctor as auditing placement, duplication, and system-prompt token use.
The public README for the Context Constitution describes it as a set of principles governing context management and as a living document. Letta says it uses those principles internally as a foundation for prompting and memory-model training, and presents active context management and retained token-space representations as an alternative to learning through model-weight updates. This is a vendor-authored account of the intended design, not empirical proof that a particular agent retrieved, noticed, or followed a retained representation.
Trace transformations between events
A chronological log records order but may omit the lineage connecting a source message to a memory unit, summary, prompt, and answer. MemTrace records variables, operations, and dependency edges in a directed execution graph. Variables include artifacts such as raw messages, memory units, summaries, and prompts. Operations include model inference, retrieval, filtering, parsing, and tool invocation. The edges record which variables each operation consumes and produces.
A production adaptation should preserve that causal skeleton. Give each source artifact, write result, retained revision, retrieval result, context snapshot, generation, and external effect an identifier. For each transformation, record input identifiers, output identifiers, operation type, timestamp, configuration version, and status. Sensitive values may need controlled storage, but the trace still needs enough identity and provenance to reconstruct the path.
Exact localization remains difficult even with structured traces. In the MemTrace arXiv v3 preprint, Table 1 reports a best overall faulty-operation identification accuracy of 46.25% for MemTrace-OBS with GPT-5.4 on MemTraceBench’s 160 annotated failures across Long-Context, RAG, Mem0, and EverMemOS. That figure describes one method, model backbone, benchmark, and collection of memory systems; it is not a general estimate of diagnostic accuracy. Treat an automatically assigned operation as a lead and require artifact-level evidence before accepting it.
Repair the failed stage
Keep repairs stage-specific. Source failures require correcting the task premise or evaluation evidence. Write failures require changing extraction, update, deletion, or consolidation behavior. Representation failures require repairing schema handling, revision selection, persistence, or synchronization. Retrieval failures require changes to queries, indexing, ranking, or filtering. Placement failures require changes to context assembly, ordering, conflict handling, or truncation. Utilization hypotheses require controlled comparisons with the relevant context held fixed or deliberately varied. Effect failures require repairing delivery or tool execution.
Avoid compensating for a retrieval defect by pinning every fact into the prompt. That can conceal the original defect while increasing placement pressure. Rewriting stored memory likewise cannot repair a delivery adapter that executes something different from the generated instruction.
After the repair, rerun the frozen incident from a known revision and inspect the actual effect. Add an adjacent case, a conflicting-memory case, and a later-update case when those paths exist. A saved file, passing unit test, or plausible answer is an intermediate receipt. The completed check follows the information into the delivered response or state mutation.
Keep an attribution record
Record the source evidence, expected and actual effects, earliest observed divergence, every operation still implicated, controlled intervention, repair, revision, rerun receipt, and residual uncertainty. If the evidence does not establish the earliest divergence or excludes additional faults, leave the incident unresolved or classified only at the broader stage. A broad diagnosis supported by artifacts is more useful than a precise label the trace cannot justify.