AI Remembers You—Why Does It Still Fail to Understand You?

For modern AI agents, parsing explicit statements like “my knee hurts” or “I only slept five hours” is now table stakes. Drawing from our work at Speediance building intelligent fitness agents, the real challenge lies in more ambiguous judgments: a user still wants to build strength, but their recovery state has changed; they say they “feel pretty good,” yet workout completion has been falling; device data says recovery is normal, but subjectively they cannot get themselves going.

None of these situations can be solved by simply “remembering a sentence.” An Agent must combine signals from conversation, training behavior, devices, and real life; decide which are facts, which are temporary states, and which are only hypotheses awaiting validation; and then determine whether to maintain the current course, adjust it, or ask for confirmation first. This is where long-term memory creates value: not by making the Agent behave like someone who has memorized the chat history, but by enabling it to continually maintain a set of decision conditions that change as the evidence changes.

We are sharing these lessons with product and engineering teams building Agents intended for long-term use. The article follows one decision chain: information first enters storage as traceable evidence; evidence then updates and maintains the user's overall state; when memory is invoked, the system compiles the context required for the current task; after the user reports the outcome of an action or corrects information, the system returns to the next round of judgment.

It is worth emphasizing that not every Agent system needs full long-term state memory. For one-off Q&A, products with a single authoritative data source, or products whose outputs do not affect later decisions, model context, structured fields, and ordinary retrieval are often enough. Full long-term state infrastructure becomes worthwhile only when historical data must be repeatedly reinterpreted, states continue to change, and the results of actions need to influence the next decision. The dividing line is whether the system must take responsibility for a process that evolves over time. Health and fitness is a representative use case—and one with especially high requirements for reliability.

Memory State Maintenance: From Information and Evidence to an Updatable User State

Once a system must take responsibility for a process that changes over time, its first requirement is the ability to continually maintain memory state. Only then can it judge what the same piece of information means in each user context. The same sentence from a user may lead to completely different Agent decisions under different circumstances.

Suppose a user says, “Next week I can only train twice.” If this is caused by a one-time business trip, the Agent only needs to compress that week's schedule and return to the original rhythm afterward. But if memory shows that the user has completed only two sessions per week throughout the past month, their available time has probably changed in a lasting way. The Agent should redesign the baseline weekly plan instead of continuing to generate four sessions and waiting for the user to repeatedly miss them.

On the surface, the system remembers that the user said they could “only train twice.” What actually determines the next action is whether that statement represents a temporary exception or a new real-world condition. Long-term memory must maintain exactly this kind of judgment—one that changes over time as behavioral evidence accumulates.

To make that judgment, the system must maintain three types of information that play different roles in decision-making:

• Relatively stable boundaries: training background, long-term goals, injury history, commonly available equipment, and other information that changes infrequently but constrains the solution.
• Changing state: sleep, pain, stress, schedule, training phase, and current recovery—information that is valid only for a limited period.
• Continuously generated behavioral evidence: completed sessions, skipped sessions, regressions, load changes, subjective feedback, and restarts. These signals show whether the plan is actually being carried out.

These categories serve different roles in the system. Some directly define long-term boundaries, some describe the current phase, and some only verify whether the first two still hold. Their credibility, validity periods, and scopes therefore differ, so the threshold for writing them into memory cannot be the same.

We use a layered state service to manage memories with these different scopes. In this process, the model is responsible only for extracting candidate memories from new input and proposing a scope. The state service then decides whether each item should remain in the evidence layer, be promoted into a stage-level state, or be used only as a temporary decision condition for the current task.

We organize memory into three connected layers that can still be corrected independently:

• Evidence layer: records what happened—for example, a workout that was abandoned, a sleep measurement, or a piece of subjective feedback—while preserving its source and time.
• State layer: gives a stage-level interpretation to a body of evidence, such as “completion of Friday-evening workouts has remained low over the past four weeks.” A state has a scope and confidence level; it is not the same as the underlying facts.
• Decision layer: generates the conditions that will actually be used for the current task, such as “do not schedule a high-friction workout this Friday; keep a flexible make-up window.” It is valid only for the relevant task.

The evidence and state layers together make up the user's long-lived maintained state. The decision layer is not persistent memory itself, but an ephemeral runtime projection compiled from the state.

A typical scenario shows how layered memory changes both memory intake and memory use.

• Suppose a user skips two workouts in a row, and both workouts happen to be relatively intense. The Agent may form the hypothesis that “the user dislikes high training intensity.”
• If the Agent writes that hypothesis directly into long-term memory, it immediately expands into a stable preference and begins to affect workout composition, reminders, recovery advice, and many downstream functions. The user is likely to experience this as an overcorrection.
• If the hypothesis enters a layered memory service, the system can recognize that it is supported by only two behavioral observations and should not immediately become long-term memory. It can remain a low-confidence, stage-level hypothesis: “the user may currently be avoiding high-intensity training; the reason still needs confirmation.” This keeps the guess at a short-lived hypothesis level instead of promoting it into a global preference.

With this three-layer memory system, we can continually maintain and revise judgments without discarding the original records. If the Agent later learns that the two skipped workouts occurred because the user was on vacation, rather than because of an intensity preference, the state layer can be updated and applied going forward. The two original skip events still remain as evidence, and any decisions or plans affected by them can be traced back to why they were changed at the time.

At this point, memory can be continually corrected by new evidence. But that solves only whether a state can change; the system must also decide when the state is still valid. Memory state therefore needs to be bound to time.

A statement like “I'm very tired today” may matter only today, while several weeks of insufficient sleep may be enough to change the current training cycle. A past surgery may remain relevant for years, but its effect on today's exercise selection still needs to be judged against the user's recent functional capacity. The system should at least distinguish current state, stage-level state, and stable traits, while recording event time, observation window, last validation time, and explicit expiry triggers. A universal TTL can make content expire, but it cannot express business meaning such as “the user confirmed the project has ended, so the related state should become invalid immediately.”

Even when the time range is clear, evidence from different sources may still conflict within the same period. The system should not eliminate that conflict by overwriting one semantically similar item with another. It should determine which source is more trustworthy for which kind of question—or combine the sources into a more complete judgment. State conflict should not be resolved through simple semantic overwrite.

For exercise feedback, for example, we can observe the user through multiple sources. Conversation best captures subjective experience. Behavioral data provides repeated samples of what the user actually does. Speediance devices add physical measurements of the behavior. For repetitions and load, we place more trust in device measurement; for today's fatigue and pain, the user's own experience must take priority; for long-term exercise preferences, we need to observe repeated behavior.

If a user says they feel good today, but training completion is falling and movement speed is slowing, both sides of the evidence should be preserved, along with the judgments each supports. Perhaps recent sleep has restored the user's perceived alertness, while accumulated physical fatigue has not yet cleared. Conflicting data is not necessarily dirty data; it is often the very signal that a user’s underlying state is shifting.

Memory Retrieval: Context Must Be Compiled into Input for the Current Decision

By this point, the system has a user state with sources, time, confidence, and conflict relationships. But maintaining state correctly does not mean the entire state should appear in every inference. For a specific task, the system must retrieve only the parts that can actually change the current decision.

As is widely understood, because attention is limited, placing more irrelevant recalled content into the context can make the judgment worse.

Suppose that during a two-week business trip, a user repeatedly asks for workouts to be compressed to twenty minutes. The system remembers this. Six months later, when the user is at home and planning a new hypertrophy cycle, semantic retrieval may still rank those “short workout” records highly because they are semantically similar to “plan this week's workouts.”

If the system does not first check validity period and task scope, it may place those historical constraints into context. The model can then easily interpret a temporary limitation as a stable preference and let it shape the new plan. The problem is not retrieval alone; the way the context is assembled misleads the system into treating the information as “necessarily applicable now.” Unnecessary context must be identified and removed, or it becomes interference.

Before every answer, plan, or proactive task, the system therefore needs to compile a small, task-specific profile: which parts of history could change this decision, which information has expired, and which risks must be checked first. We call the module that performs this work the Context Compiler. It behaves more like a compiler than a search box: it translates distributed, heterogeneous user state into the constraints and evidence that the model can execute in this round.

A Context Pack usually goes through four steps:

• First, apply deterministic filtering by permission, type, validity period, and task scope.
• Then use semantic retrieval to recover potentially relevant candidates.
• Next, rerank by recency, confidence, decision impact, and conflict relationships.
• Finally, trim the result to the best context window for model comprehension.

We give priority to the current goal, current plan, and risk boundaries. Historical cases and weakly relevant preferences are loaded only when needed. The point is not merely to save a few tokens; more importantly, it reduces the interference that irrelevant memories create for model judgment.

Closing the Memory Feedback Loop: Outcomes, Versions, and User Corrections

Filtered and compiled context enters the model and produces advice, a plan, or a proactive task. But this is still only one short-term decision. Many health goals require observation over time: improving sleep may take a month of behavioral adjustment and review; recovery management requires training load, sleep, subjective fatigue, and completion to be placed on the same timeline. Continually correcting memory through ongoing feedback is therefore essential.

To connect a decision with its later outcome, the system must keep recording what it is trying, what it expects to observe, and when it will review the result. A health thread provides this continuity across many conversations and actions. It records the goal, current phase, strategies being tried, expected indicators, recent changes, unresolved questions, and the next review time. Behind conversation as the interaction entry point, the status, plans, and observation conditions of a series of health-management tasks can therefore persist.

The most important requirement for continually maintaining the information and strategies in a health thread is to use user feedback to keep correcting its decision state.

User feedback here includes more than information sent in conversation. Active controls—viewing, editing, pausing, or deleting information—are also high-quality behavioral inputs, because they often tell the system directly, “this is how I want you to understand me.” Long-term memory forms a real closed loop only when every type of interaction in the system, including activity data and outcomes, conversational feedback, and control actions, can change the next state.

Behavioral feedback must also be interpreted correctly. If a user rejects a workout reminder once, they may dislike the timing of the reminder; it does not necessarily mean they no longer care about training. Our approach is to keep memory about interaction strategy separate from memory about health and exercise preferences. In the short term, this feedback enters interaction-strategy memory. Over time, the system continues to observe whether it should be promoted into exercise-preference memory.

At this point we have done considerable work to interpret an individual piece of feedback correctly, but that is still not enough. Whenever feedback changes the state of a user's health thread, plans, reminders, and proactive tasks generated from the old state must also be updated. To understand how one piece of feedback affects the whole system—which data, tasks, and states it reaches—we need data lineage that can trace and evaluate the impact of outdated information end to end.

Suppose a user has denied that they “prefer evening workouts.” The profile shows the correction, but a reminder task generated from the old context is still running and reproducing itself. A few days later the Agent sends another evening reminder, which is obviously confusing. Correcting the information properly requires following its data lineage to find every hypothesis, plan, and proactive task that the old state once supported, then marking those downstream results as invalid, pending review, or in need of regeneration. Only when the new feedback becomes evidence, the old state is replaced by a new version, and the related downstream results are invalidated will the next interaction truly make the correction visible to the user.

Only then is the decision chain driven by user feedback fully closed, able to keep adapting its judgments and states with every new round of feedback.

Conclusion: The Goal of Memory Is Not to Recall More, but to Make Better Decisions Over Time

Seen as one system, long-term memory becomes a continually updated decision loop rather than a storage plugin attached to the model:

For a health and fitness Agent, the competitive advantage of long-term memory is not how much user information it stores. It is whether the system can organize scattered life signals into a user state that is updatable, explainable, and correctable—and use only the truly relevant parts at each moment. In architectural terms, this is not a retrieval plugin but state infrastructure: events are traceable, judgments are versioned, recall has scope, conflicts are preserved, and user corrections can propagate downstream.

When user goals, physical state, real-world schedule, and past actions are placed on one continuous chain, the Agent can move from “giving a recommendation” to “maintaining a process together with the user.” A mature implementation is not necessarily magical: the model interprets and weighs evidence; the state service decides what to remember; the rules layer protects permissions and hard boundaries; and the Context Compiler decides what the model should see in the current round.

Together, these responsibilities ensure that every judgment can be traced to evidence, new feedback can correct old state, and the next action can move closer to what the user actually needs now.