# StateBench tests how well language models track state changes over long documents > Frontier models drop to 13% accuracy or lower on the hardest state-tracking problems in StateBench, a new benchmark that tests how well AI follows changing variables across long documents. - Type: AI news - Page: https://completeaitraining.com/news/statebench-tests-how-well-language-models-track-state/ - Published: 2026-11-04 - Updated: 2026-11-04 - Author: Jeroen Erne - Jobs: Education, IT and development, Product development, Science and research, Writers - Reading time: 3 min Tracking who has the knife, what variable holds the count, or where the ball is under the cup - these are state-tracking problems that push language models to their limits. A new benchmark called StateBench, developed by researchers working on long-context model evaluation, measures how well AI systems follow information about entities as they change across long documents, and the results expose clear weaknesses even in frontier models. #### Four levels of difficulty StateBench breaks state tracking into four tiers, each permitting more complex update operations. The levels draw on Krohn-Rhodes theory, a branch of algebra that describes how state-tracking problems can be decomposed into atomic primitives. **R-trivial** problems use immutable assignment - a variable's value never changes, so the model only needs to recover what it was first set to. In everyday terms, it is like a birth date that stays fixed. **Aperiodic** problems allow reassignment by value; the model must find the most recent value and can ignore earlier ones, like a thermostat where the latest setting determines the current temperature. **Soluble** problems add self-referential updates where each change builds on the current value. The model has to carry changes forward rather than simply find the latest assignment, much like a game score where each point adds to the running total. **General finite** problems, the hardest tier, add assignment by reference - one variable can take on another variable's current value, creating dependencies across updates. The researchers illustrate this with a magician hiding a ball under one of five cups and repeatedly swapping their positions. Following the ball means tracking how those swaps change its location. To compare the four levels fairly, each test starts with a general finite sequence of variable assignments and derives three easier versions from it. All four use the same number of variables and distinct values, with assignments appearing at the same points in the sequence; only the type of assignment changes. #### Where pretrained models break down When tested on open pretrained models - including two 7-billion-parameter Olmo 3 variants (a transformer and a hybrid that adds Gated DeltaNet layers) along with smaller 275-million and 2.7-billion-parameter models - almost all performed well on r-trivial problems. Accuracy dropped on aperiodic tasks as inputs grew longer, and all models stayed at chance on general finite problems. None of these models had seen contexts longer than 64,000 tokens during training, and most were tested on inputs well beyond that length. Problem format mattered. The 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on r-trivial problems presented in a RULER-style format but were near chance on the same problems framed as spreadsheet edits. That gap is one reason StateBench tests the same underlying problem in several formats. #### Frontier models stumble on harder tasks The researchers tested four frontier models - Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna - asking them to read each sequence of variable assignments and generate the final value of every variable. All four performed well on r-trivial and aperiodic problems. But on soluble and general finite problems, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems. One likely factor is sensitivity: at these harder levels, many more steps can affect the final answer, leaving less room for a model to miss an update without getting the result wrong. #### Direct training reveals architecture limits To separate architectural limits from what pretraining simply does not teach, the team trained several 275-million-parameter models from scratch on StateBench - two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. The models trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments, or 250 times the maximum training length. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences tested. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite problems. On the hardest level, the Gated DeltaNet variant with a more flexible memory update generalized the farthest, staying near perfect out to roughly 10 times the length of the longest sequences it saw during training. For Gated DeltaNet and hybrid models, the gap points to training rather than architecture - trained directly, they learned to handle state-tracking levels their pretrained counterparts could not. Transformers were different: even trained directly, they could never learn the hardest level. #### Why this matters for developers and researchers State tracking underlies tasks from reading code - where variables are assigned and reassigned - to following a story as characters physically move around and objects change hands. If a mystery novel states that "John gave Mary the knife" right before a murder occurs, knowing that the state of each character has changed is essential. As recurrent and hybrid architectures like Gated DeltaNet become more common, understanding which kinds of state tracking each design can handle will shape how future systems are built and trained. StateBench gives teams a finer-grained tool to find where models break down and to build training and evaluation data for the capabilities that still fall short. For those looking to deepen their understanding of these model behaviors, [AI Research Courses](https://completeaitraining.com/tag/research/) cover evaluation methodologies relevant to this kind of analysis. --- This is the Markdown form of https://completeaitraining.com/news/statebench-tests-how-well-language-models-track-state/. The whole site is mapped at https://completeaitraining.com/llms.txt.