Language models track evolving facts by routing internal representations backward

Feeding deep activations back into early layers allows frozen artificial intelligence models to track changing context, cutting errors without retraining or extra thinking tokens.

When a reader follows a narrative, an interpretation settled in one sentence stays active for the next. A person who reads about someone walking to a riverbank with a fishing rod does not wonder whether that bank has an automated teller machine when the word appears again moments later. The surrounding context resolves the ambiguity, and that conclusion quietly guides how every subsequent sentence is understood.

Standard artificial intelligence systems struggle with this basic form of continuity. In a conventional feedforward transformer, information travels in a single direction, moving upward through stacked computational layers from initial word input to final prediction.¹² While deeper layers successfully resolve ambiguous meanings using the full prompt, those refined conclusions are discarded when the network begins reading the next word.¹² The model starts each successive step from scratch, leaving it vulnerable to confusing riverbanks with financial institutions.

To understand why this happens, consider how a transformer processes sequential text. When an ambiguous word first enters the network, early computational layers activate representations for several possible meanings.¹ As processing ascends through deeper layers, attention mechanisms pull in surrounding context such as fishing gear or flowing water, allowing the model to settle on the correct interpretation.²¹ Yet because the architecture only carries information forward, the shallow layers that receive the next incoming word never see that settled interpretation.²³ A research team led by Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, and Rosanne Liu at Google DeepMind and the University of Texas at Austin has introduced an alternative called recirculation. In a preprint (which may not have been peer reviewed) posted to arXiv in August 2026, the researchers describe a method that routes internal signals from deep within a network back to its shallower layers.⁴⁵ This feedback loop allows an off-the-shelf model to behave like a dynamical system that carries its internal belief state forward from one moment to the next.⁴⁶

How does recirculation pass information backward through a transformer?

Recirculation routes a normalized activation vector from a selected deep layer back into a shallower layer as the network processes the next input step.¹ Because the underlying residual stream shares a common feature space across different depths of the network, the deeper representation can enter the shallower layer directly without requiring a learned adapter.²⁶ This feedback mechanism creates a direct recurrence loop across depth and time, allowing contextualized states to persist across successive tokens without altering any underlying model parameters.⁵⁶ Unlike looped transformers that repeatedly execute layers within the same token step to increase computational depth, recirculation links distinct time steps together.²

Achieving this balance requires careful calibration of how much signal returns to earlier stages. Michael C. Mozer, a lead author on the paper, told Primary that the central technical challenge lies in managing the mixing magnitude, explaining that "the representations are pushed out of distribution if the top-down feedback is too strong."^contributed Mozer said the objective is "to balance the benefit of making potentially relevant information available to shallower layers and the cost of changing the representations in the shallow layers so much that the model no longer processes them properly," noting that the team tested whether they could find a balanced mixing volume.^contributed In practice, the technique rescales the deep activation vector to match the destination layer's magnitude before taking a convex combination governed by a fractional coefficient.¹

The researchers evaluated two distinct versions of the technique while keeping the core network frozen. In basic recirculation, the choice of source layer, destination layer, and mixing proportions is fixed through an initial search without updating any parameters.^contributed Mozer told Primary that the research team "conducted an exhaustive sweep of (source, destination) layer pairs to better understand a model's natural proclivities," observing consistent perplexity reductions across robust regions of the pairing space.^contributed In adaptive recirculation, the base model remains frozen while a small multi-layer perceptron learns to calculate token-conditioned mixing coefficients based on incoming activations.¹⁴ Analysis published on explainx.ai by Yash Thakker noted that while basic recirculation is strictly training-free, the adaptive variant preserves base weights while training that auxiliary network on 250 documents from each of arXiv, C4, and PG19.¹ Asked about that training dataset, Mozer told Primary that the 250-document selection "was an arbitrary choice for the purpose of a demonstration" to show what could be achieved with minimal optimization.^contributed

What performance gains does internal feedback achieve across benchmarks?

Feeding deep activations back into early layers reduces perplexity across language modeling suites and improves performance on multi-step reasoning tasks.⁴⁸ With out-of-distribution inputs, the Gemma 3 model family can obtain reductions in perplexity of 10 to 15%.² In testing on held out datasets, perplexity reliably drops on 10 of 10 datasets.⁶⁵ Importantly, perplexity drops most on tokens which require a long context history to predict well, as reflected in a measure called LongPPL.⁶

On mathematical reasoning, the preprint reported a 21% accuracy gain on the GSM8k benchmark.⁴¹ As Yash Thakker reported on explainx.ai, that headline represents a 20.9% relative reduction in error rate when sampling 128 potential answers on Gemma 3 4B, alongside an 8.8% relative error reduction on the greedy first answer.¹ Clarifying the underlying figures, Mozer told Primary that "absolute pass@1 for the Gemma3 4B model was 29.3%, recirculation achieved 30.6%, and adaptive recirculation achieved 35.5%."^contributed Mozer added that because the benchmark uses a single test set and deterministic greedy evaluation, those deterministic baseline and intervention scores carry no variance across repeated runs.^contributed Gains on eight single-token multiple-choice benchmarks were more modest and varied across datasets.¹⁵

What are the operational limits of sequential prefill?

The primary operational limitation of recirculation is that prompt ingestion during the prefill phase runs sequentially rather than in parallel.⁴¹ In standard transformer architectures, all prompt tokens can be processed simultaneously during prefill, but feeding deep activations forward across time requires each token step to finish before the next can begin.²¹ While the authors reported essentially no added latency during text generation on parallel hardware stacks, the serialization of prefill creates a computational bottleneck on long documents.⁴¹ Mozer noted to Primary that evaluating this trade-off requires weighing downstream accuracy gains directly against the computational cost of sequential prefill, and noted that perplexity served primarily as an experimental tool to discover where recurrence belonged.^contributed

The findings also represent research measurements on specific architectures rather than universal production rules. Testing across other model families, including Ministral 3, Pythia, Qwen 3, and Phi-2, showed less than 0.5% perplexity reduction under limited tuning, compared with roughly 5% on comparable Gemma setups.¹ Mozer told Primary that "most tokens should not benefit much from being recirculated; after all, the transformer was trained to perform well with purely feedforward processing."^contributed The results come from a version-one preprint that has not undergone peer review, and optimal layer choices and mixing parameters remain sensitive to specific model architectures and domains.² Addressing whether the mechanism might simply act as random regularization, Mozer told Primary that the researchers "did run studies where we noised up representations, played with softmax temperatures, etc. to see if any of these manipulations could mimic recirculation," without finding any that did, while targeting specific tokens that need state tracking yielded larger perplexity improvements.^contributed

How does internal state tracking change model design?

Recirculation provides machine learning engineers with a third operational lever between prompt engineering and full model retraining. By spending compute on maintaining continuous internal representations, models can track evolving states without relying exclusively on chain-of-thought prompting, which emits verbose textual reasoning steps that consume output bandwidth and increase generation costs.⁴¹ While explicit reasoning chains remain valuable for complicated multi-step deductions, maintaining an evolving internal belief state relieves the model from having to verbalize every minor contextual update into output text.²¹

The approach indicates that trained foundation models contain latent architectural pathways that can be activated after initial training completes.⁴¹ Mozer told Primary that the broader contribution lies in "understanding model affordances (what the model wants to do naturally without us forcing it to perform a certain way)," focusing on what trained networks naturally support without forcing structural redesigns from the outside.^contributed Future work will determine whether block-based processing can alleviate the prefill bottleneck on lengthy contexts, and whether recurrence paths can be discovered automatically across broader model families.

What this rests on

40 sentences trace to 7 sources and 1 contributor.

All 8Documents 2Records 0Reporting 5People 1

1. 1 E explainx.ai Third-party reporting · explainx.ai · 24 Aug 2026 DeepMind Recirculation: The 21% GSM8K Claim Explained | explainx.ai Blog See the source
2. 2 A arxiv.org Paper · arxiv.org Recirculation Preprint · may not have been peer reviewed See the source
3. 3 N note（ノート） Third-party reporting · note.com · 25 Aug 2026 (260825)[Fable5 Dialogue] To Carry It Inward Is to Sediment It, and to Make It Invisible — Recirculat｜יודה See the source
4. 4 MC Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu Paper · arxiv.org · 31 Aug 2026 Recirculation Preprint · may not have been peer reviewed See the source
5. 5 D daily.dev Third-party reporting · daily.dev · 25 Aug 2026 Google DeepMind's "Recirculation": Feeding Activations Back Through a Frozen Transformer | daily.dev See the source
6. 6 L linkedin.com Third-party reporting · linkedin.com · 20 Aug 2026 Recirculation Boosts Accuracy and Reduces Perplexity in Foundation Models | Nishantha Ruwan posted on the topic | LinkedIn See the source
7. 7 MC Michael C. Mozer, lead author Contributor · interview 2 Sep 2026 · in the article since 24 Sep 2026 Contribution — Michael C. Mozer 16 statements added to this article
8. 8 C cryptobriefing.com Third-party reporting · cryptobriefing.com · 25 Aug 2026 Google DeepMind paper reveals recirculation method improves transformer context handling See the source

Also consulted: 8 pages across 6 sites, none cited.

Article history

1. Published 24 Sep 2026, 12:41
   Assembled by the Primary desk from 7 sources · 1 contributor · 40 cited sentences
2. MC Michael C. Mozer, lead author 16 statements added 2 Sep 2026 23 days before publication

   1. Go to the sentence: Michael C. Mozer, a lead author on the paper, told Primary that the central technical challenge lies in managing the mixing magnitude, explaining that "the representations are pushed out of distribution if the top-down feedback is too strong."↑
   2. Go to the sentence: Mozer said the objective is "to balance the benefit of making potentially relevant information available to shallower layers and the cost of changing the representations in the shallow layers so much that the model no longer processes them properly," noting that the team tested whether they could find a balanced mixing volume.↑
   3. Go to the sentence: In basic recirculation, the choice of source layer, destination layer, and mixing proportions is fixed through an initial search without updating any parameters.↑
   4. Go to the sentence: Mozer told Primary that the research team "conducted an exhaustive sweep of (source, destination) layer pairs to better understand a model's natural proclivities," observing consistent perplexity reductions across robust regions of the pairing space.↑
   5. Go to the sentence: Analysis published on explainx.ai by Yash Thakker noted that while basic recirculation is strictly training-free, the adaptive variant preserves base weights while training that auxiliary network on 250 documents from each of arXiv, C4, and PG19.↑
   6. Go to the sentence: Asked about that training dataset, Mozer told Primary that the 250-document selection "was an arbitrary choice for the purpose of a demonstration" to show what could be achieved with minimal optimization.↑
   7. Go to the sentence: With out-of-distribution inputs, the Gemma 3 model family can obtain reductions in perplexity of 10 to 15%.↑
   8. Go to the sentence: Importantly, perplexity drops most on tokens which require a long context history to predict well, as reflected in a measure called LongPPL.↑
   9. Go to the sentence: As Yash Thakker reported on explainx.ai, that headline represents a 20.9% relative reduction in error rate when sampling 128 potential answers on Gemma 3 4B, alongside an 8.8% relative error reduction on the greedy first answer.↑
   10. Go to the sentence: Clarifying the underlying figures, Mozer told Primary that "absolute pass@1 for the Gemma3 4B model was 29.3%, recirculation achieved 30.6%, and adaptive recirculation achieved 35.5%."↑
   11. Go to the sentence: Mozer added that because the benchmark uses a single test set and deterministic greedy evaluation, those deterministic baseline and intervention scores carry no variance across repeated runs.↑
   12. Go to the sentence: Mozer noted to Primary that evaluating this trade-off requires weighing downstream accuracy gains directly against the computational cost of sequential prefill, and noted that perplexity served primarily as an experimental tool to discover where recurrence belonged.↑
   13. Go to the sentence: Testing across other model families, including Ministral 3, Pythia, Qwen 3, and Phi-2, showed less than 0.5% perplexity reduction under limited tuning, compared with roughly 5% on comparable Gemma setups.↑
   14. Go to the sentence: Mozer told Primary that "most tokens should not benefit much from being recirculated; after all, the transformer was trained to perform well with purely feedforward processing."↑
   15. Go to the sentence: Addressing whether the mechanism might simply act as random regularization, Mozer told Primary that the researchers "did run studies where we noised up representations, played with softmax temperatures, etc. to see if any of these manipulations could mimic recirculation," without finding any that did, while targeting specific tokens that need state tracking yielded larger perplexity improvements.↑
   16. Go to the sentence: Mozer told Primary that the broader contribution lies in "understanding model affordances (what the model wants to do naturally without us forcing it to perform a certain way)," focusing on what trained networks naturally support without forcing structural redesigns from the outside.↑