Your AI read everything. It still lost the plot.

You tell an AI assistant the launch is Friday. Then Monday. Then that the old Friday checklist is obsolete. Twenty messages later, it confidently prepares your Friday launch.

Everything was in the conversation. So why is the assistant organising a party for the wrong day?

That is the uncomfortable distinction behind Ai2’s StateBench announcement on 3 November 2026: having room for information is different from correctly tracking what that information has become.

We followed the research with a small test of five models. Four produced tidy answers that lost track of changing values. The fifth spent its answer allowance explaining the task and never delivered the requested result.

A bigger filing cabinet is useful. It does not automatically come with a better bookkeeper.

Read it. Find it. Keep it straight.

A context window is the amount of material a model can work with in a request. Its advertised size is a capacity specification, not a guarantee that every relationship inside it will be handled correctly.

Finding a sentence is one job. Working out which sentence supersedes another is a different job. Following a sequence in which each change depends on the previous state is harder again.

StateBench separates four kinds of tracking: fixed facts, replacement values, running updates and changes that depend on other variables. Ai2 reports that its tested frontier models deteriorated on the harder tasks as sequences grew longer. In experiments with 275-million-parameter models trained from scratch on these tasks, some recurrent and hybrid designs generalised farther than transformers. That is an encouraging research direction, not evidence that every hybrid assistant already has reliable memory. Read Ai2’s findings.

The released benchmark code derives easier tasks from a common underlying trace. This matters: it helps separate operation complexity from changes in the test itself. Our demonstration below uses independently generated sequences and does not reproduce that controlled architecture comparison.

Think of a coding assistant. It may find the original function, quote the later patch and still reason from an assumption that patch removed. Or consider a project chat where a feature was proposed, parked, approved and finally cancelled. Mentioning all four messages is not the same as knowing the feature is cancelled.

Those are practical analogies. We did not measure real coding projects or long conversations in this pilot.

Five variables. No hiding place.

Our test began with five variables, A through E, holding 0, 1, 2, 3 and 4. Each prompt contained 24 numbered operations and asked for the final values as JSON, with no explanation.

The first task replaced values. The second added numbers and kept the remainder after division by five. The third swapped current values between variables. Every model received the same three saved prompts, including the same instruction that swaps happen simultaneously.

We used local Qwen 3 4B, DeepSeek Flash, GPT-5.4 Mini, Claude Haiku 5.5 and Gemini 3.1 Flash-Lite. These were accessible fast-response models, not a scientifically matched capability group. A quantised local 4B model and proprietary cloud models are very different things.

There were no tools, web searches, answer schemas or retries. Each request had a 160-token output allowance. Explicit thinking was disabled where configured; Gemini used its minimal setting. The exact controls and model identifiers are in the methods.

Crucially, these prompts were short. This was a StateBench-inspired demonstration of following changes, not a test of million-token windows, official StateBench scores or what happens when a model runs out of context.

The JSON looked splendid. The answers didn’t.

Qwen, DeepSeek, OpenAI and Gemini each got all five replacement values right. Each then failed to return a fully correct result for either of the two harder prompts.

On the running-update task, the expected final state was A=3, B=0, C=3, D=4, E=3. GPT-5.4 Mini returned A=0, B=4, C=4, D=0, E=4. All five fields were wrong. The response was complete, valid JSON and available in 1.24 seconds.

That is the awkward part of automation: a well-formed answer can sail through a format check while failing the actual job.

Qwen finished with eight correct fields across the three tasks; DeepSeek and Gemini seven each; OpenAI six. Those tiny totals are observations, not a league table. One extra correct variable in one attempt is not evidence that a model is generally more intelligent.

Claude found a different way to miss the exit

Claude Haiku 5.5 did not return the requested JSON. It began explaining the operations and used all 160 output tokens on every attempt. Each response ended with max_tokens before a final answer appeared. Reported thinking-token usage was zero.

Our strict scorer counted no completed answers. But calling that “15 arithmetic mistakes” would be wrong. We do not have the final calculations to judge. What we observed was an instruction-following and completion failure under this answer budget.

A larger allowance, different prompt, structured output or different thinking configuration might change the result. We did not test those conditions, and silently giving one model another go would have changed the comparison.

Stop asking the transcript to be the database

The everyday lesson is not to abandon large context windows. They are useful. The mistake is treating capacity as a substitute for verified current state.

For a long project, keep a short ledger of what is true now: current decisions, superseded decisions, unresolved questions and the source of each change. Update it deliberately. A model-generated summary needs checking too; otherwise you have simply promoted yesterday’s mistake into today’s official record.

For code, read the current files and run the relevant tests. For arithmetic, inventory or balances, use deterministic calculations and validate the result. Ask the model to explain the verified state, rather than making its prose the sole source of truth.

For changing plans, retrieving the latest relevant decision may help. For a chain of updates, the latest line alone may be insufficient: you need the starting state and the changes that lead from it.

Product teams should test these jobs separately. Can the assistant find a fact? Can it identify an obsolete instruction? Can it compute the new state? Can it return a usable answer within the configured limit? One impressive context-window number answers none of those questions by itself.

The small print that makes the big claim smaller

This pilot used one independently generated prompt per task, one attempt per model and 24 operations per prompt. There were 15 model requests in total, run on 4 November 2026. The sequences differed between task types, so their scores do not isolate difficulty as cleanly as the official benchmark’s matched traces.

Expected values were computed by forward simulation, cross-checked using a second method, then audited again from the saved prompt text. A field passed only if its returned integer matched. A whole prompt required the complete expected JSON object. Markdown fences were tolerated consistently; malformed or incomplete answers were retained.

These were direct API and local model requests, not a test of the consumer chat apps, their memory features or a tool-using agent. No private conversation history or expected answers went to the models. No repeats, reasoning-enabled comparison or context-length sweep was performed.

Download the exact prompts and expected answers · Download the selected responses and scores

Ai2 has released StateBench’s dataset and code. Its framework uses a fixed set of variables; real conversations can introduce new people, objects and rules. Neither that research nor this demonstration is a complete simulation of everyday assistant memory.

Conclusion

AI can digest remarkable amounts of material. What matters next is whether it can preserve the meaning of the changes inside it.

Our small experiment does not establish a winning provider or a broken architecture. It does show why a correct format, a fast response and access to the full prompt are not enough to establish a correct answer.

Keep the ambition for larger contexts. Demand evidence that the assistant knows which decision is current, which assumption was cancelled and which values changed.

Before asking how much your AI can read, ask it to prove it knows what is true now.