Language models compress their memory by tracking how attention mixes tokens

Retaining vector directions based on the mixed attention output rather than isolated token energy yields an exact mathematical compression rule for inference memory.

When a large language model generates a long passage of text, it must remember every preceding word to decide what comes next. In standard generative models, this memory is held in a workspace called the key-value cache, where every generated token leaves behind numerical records of its identity and meaning.¹ As conversations and documents stretch to thousands of words, this internal ledger swells, demanding tens of gigabytes of memory on specialized processors and slowing down generation.

Engineers and researchers have evaluated several methods to shrink this memory footprint. The most intuitive shortcut is to measure which features inside the stored values contain the most energy or variation, keeping those primary directions and discarding the rest through linear projection. Yet in practice, models compressed by standard feature projection often degrade unexpectedly on complex text generation.

The flaw in that direct compression lies in how attention mechanisms actually process information. Inside an attention layer, a model does not read stored values in isolation. Instead, it computes an attention matrix that blends and weights the value vectors across multiple preceding tokens, producing a combined output. A mathematical feature that appears dominant within a single isolated word vector can cancel out completely when averaged across a sentence, while a subtle feature that seems minor in isolation can survive that mixing to drive the final prediction.

To see how this happens, trace the sequence of operations during generation. An incoming query compares itself against all past keys to generate a set of normalized attention weights. These weights multiply the stored values, collapsing many distinct historical positions into a single mixed output vector. Only after this mixing takes place does the network project the result into subsequent feed-forward layers. Because the attention matrix filters and mixes the vectors before any downstream block touches them, importance is determined after mixing, not before it.

Why does standard dimension reduction fail on stored values?

Standard dimension reduction fails because it minimizes reconstruction error on raw value vectors rather than on the mixed attention output that downstream layers actually receive. When a model projects values into a lower-dimensional space using conventional principal component analysis, it chooses coordinates that retain the largest variance across individual tokens. If the attention weights then average those tokens together, coordinates that held high opposite values cancel each other out, leaving behind zero useful signal while smaller, preserved coordinates are lost.

In a preprint posted to the arXiv server in September 2026, independent researcher Samuel Salfati at fraQtl AI Research analyzed this exact mechanism.¹² Salfati showed through a simple two-dimensional example how standard value projection can discard the only coordinate that matters. In his constructed scenario, one direction carried nine times more squared energy than the other across two input tokens, with values of 18 and two respectively.¹ However, when an unmasked uniform attention operation averaged the two tokens together, the first direction vanished entirely to zero in the output, while the smaller second direction survived with an energy of two.¹

Under rank-one compression, conventional uncentered principal component analysis preserved the first direction because it contained most of the isolated value energy, incurring a squared output error of two.¹ In contrast, projecting onto the subspace derived from the attention-weighted output retained the second coordinate, reproducing the mixed output with zero squared error.¹ The example demonstrated that what matters to a language model is not the raw size of a stored vector, but how that vector behaves after the attention matrix mixes it across the sequence.

Reflecting on this stark divergence, Salfati told Primary that isolated value energy is simply the wrong object to optimize because it represents a confident measurement of something the model never actually reads. He explained that "the direction carrying 9/10th of the energy cancels out to exactly zero once you average the two tokens, while the direction that looks negligible in isolation is the only thing that survives the output."^contributed In such cases, uncentered principal component analysis discards the one coordinate that matters and keeps the one that vanishes.

How does attention-weighted value projection solve the problem?

Attention-weighted value projection solves the problem by computing an orthogonal projection from the eigenvectors of the attention-weighted Gram matrix rather than the raw value covariance. In Salfati's paper, the author formulated the objective as minimizing the squared Frobenius norm between the true attention output and the output formed by projecting values through a low-rank orthogonal matrix.¹ For a sequence with value matrix V and fixed attention weight matrix alpha, the optimal projection matrix is formed by the leading eigenvectors of the product of the transpose of V, the transpose of alpha, alpha, and V.¹

This formulation provides an exact closed-form solution for any chosen rank. Salfati proved that the minimum distortion achieved by retaining r dimensions is exactly equal to the sum of the discarded eigenvalues, extending the classical Rayleigh-Ritz variational principle to the attention-output objective.¹ In practice, this means the spectral tail of the matrix directly measures the local squared reconstruction error incurred by dropping dimensions.

Explaining the underlying geometry, Salfati said in response to questions from Primary that downstream layers only consume the mixed output, meaning an optimal low-rank projection must minimize error on that product rather than on raw inputs. Expanding the squared Frobenius norm reveals that the inner operator controlling reconstruction error is literally the covariance matrix of the attention outputs. In this setup, high-energy components that cancel out during token mixing get scaled to zero, instructing the compressor to drop them, while subtle features that survive mixing get promoted.

The technique also adapts to modern architectures that use grouped-query attention, where several distinct query heads share a single key-value head. When multiple query heads read from the same value states, their individual Gram matrices must be summed together with non-negative weights before performing the eigendecomposition.^contributed Salfati noted that replacing this weighted sum with the Gram matrix of the mean attention across heads introduces cross-head interaction terms that alter the mathematical optimizer, making the sum of individual Gram matrices the mathematically consistent choice.¹

Elaborating on this architectural constraint, Salfati explained to Primary that each query head possesses its own perspective on which value directions survive mixing. When formulating a combined loss across heads, summing the per-head Gram matrices allows each head to independently register its required vector directions, weighted by its own attention energy. In contrast, averaging attention weights prior to forming the matrix introduces artificial cross-product terms, optimizing for a fictional averaged head rather than the genuine multi-head ensemble.

What can the mathematical result tell us about real-world deployment?

The mathematical result guarantees optimal reconstruction for fixed calibration inputs, but it does not guarantee that minimizing local reconstruction error will minimize end-to-end task loss in a live model. Downstream layers apply non-linear activations and feed-forward transformations, meaning small local errors can compound or be magnified as signals travel through the network. Furthermore, compressing values in earlier layers alters the queries and keys computed in subsequent layers, an interaction that independent layer-by-layer projections cannot globally optimize.¹

Addressing this limitation, Salfati told Primary that independent per-layer optimization assumes attention patterns remain frozen at calibration values. While minor downstream perturbations often average out harmlessly across layers, the framework fails completely when error propagation alters attention structure enough to change which tokens a later head attends to. Under aggressive compression ratios or during long-context retrieval tasks, shifts in the residual stream cause the calibration matrix to become qualitatively inaccurate, requiring joint cross-layer optimization.

The study also examined historical weight-perturbation experiments on models including GPT-2, Llama-3.2 3B, and Mistral 7B to evaluate how low-rank projections compare against weight quantization.¹ On the WikiText-2 benchmark, rounding projection weights to four bits yielded lower perplexity scores than projecting values down to 32 dimensions. For example, in historical measurements reported for Mistral 7B, four-bit weight rounding achieved a perplexity of 9.42, whereas a rank-32 value projection gave a perplexity of 44.19, and a rank-64 projection gave 13.07.

However, Salfati clarified that these historical experiments altered static model weights and immediately reconstructed dense floating-point numbers during execution, rather than storing compact integer codes or low-rank vectors in a physical key-value cache.¹ Consequently, the reported perplexity numbers reflect weight-perturbation sensitivity rather than a true comparison between low-rank caching and activation quantization at equal memory storage.¹

Discussing the hardware reality, Salfati said that during autoregressive decoding, memory bandwidth and kernel execution speed dominate system performance rather than raw arithmetic. Dedicated integer tensor cores execute low-bit quantization efficiently, whereas low-rank projection introduces operational overhead from storing bases and performing dynamic reconstructions. Salfati noted that at matched bit budgets, pure low-rank projection tends to lose against quantization, suggesting that the true value of the output-Gram geometry lies in guiding bit allocation rather than serving as a standalone storage format.

As a preprint that may not have been peer reviewed, the paper establishes a theoretical baseline rather than an operational runtime system. The derivation assumes fixed attention weights during calibration, which act as an oracle for the specific calibration sequences but may shift when a model encounters unfamiliar prompts or tasks. Moreover, linear output projection matrices that follow the attention block can alter which error directions matter most, meaning local output optimality does not strictly translate into downstream preservation.

How do alternative methods manage the key-value memory bottleneck?

Alternative methods manage the memory bottleneck by exploiting structural differences between keys and values or combining low-rank projection with offline calibration. Earlier research by Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy at Purdue University introduced Eigen Attention, which applies singular value decomposition across keys, queries, and values to compress the cache dimension by up to 40 percent while reducing attention latency by up to 60 percent. Their analysis showed that key states often exhibit a steeper eigenvalue decay than value states, indicating that keys are intrinsically more compressible than values.⁴⁵

Building on the observation that values carry dense information across many dimensions, Jian Yuan, Ziwei He, Haoli Bai, Jingwen Leng, and Bo Jiang developed WeightedKV, an approach from Shanghai Jiao Tong University and Huawei Noah's Ark Lab.⁵ WeightedKV discards keys from less important tokens while merging their corresponding values into adjacent tokens using a convex combination weighted by cumulative attention scores.⁵ Because values maintain higher information density across their coordinates, merging them rather than deleting them preserves contextual details that standard eviction methods discard.

Addressing the latency overhead of runtime projections, Xianglong Yan, Zhiteng Li, Tianao Zhang, Linghe Kong, Yulun Zhang, and Xiaokang Yang at Shanghai Jiao Tong University proposed ReCalKV. Their method uses head-wise similarity-aware reordering to cluster similar key heads before singular value decomposition, while compressing values through offline calibration and matrix fusion. By fusing the right factor of the value decomposition into the subsequent output projection matrix, ReCalKV eliminates explicit intermediate reconstruction overhead during autoregressive decoding.⁶

In complementary work exploring learned rank assignment, Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang, SeungJae Lee, Jungwook Choi, and Mingu Kang developed STAR-KV. The authors introduced a differentiable soft-thresholding mechanism to select optimal ranks across attention heads and layers, combining low-rank factorizations with low-rank-aware mixed-precision quantization.⁷ Using custom GPU kernels, their framework achieved up to 75 percent cache compression and delivered substantial latency speedups on long sequences.⁷

What questions remain for cache compression?

The primary unresolved question is how low-rank projection and activation quantization behave when evaluated side by side in a fully implemented, causal cache pipeline at equal physical storage. In theoretical payload calculations for a cache with equal key and value dimension d, 16-bit floating-point keys and values consume 32d bits per token, while storing four-bit values alongside 16-bit keys yields 20d bits, achieving a nominal 37.5 percent reduction.¹ Moving both keys and values to four-bit integers yields 8d bits per token, representing a 75 percent reduction, but these nominal calculations omit quantization scales, projection bases, and alignment overhead.¹

Future research will require running causal benchmarks where both compressed representations execute inside dedicated memory-efficient kernels, measuring actual latency and throughput during long-context generation. Researchers must also establish whether dynamic rank allocation across layers can account for downstream non-linearities and error propagation through deep networks. Until full activation-level benchmarks are conducted, the choice between discarding dimensions and reducing precision remains an open engineering trade-off.

This piece was prepared from the arXiv preprint and public records together with answers from Samuel Salfati to six questions from the Primary news team, completed September 2026.

What this rests on

24 sentences trace to 6 sources and 1 contributor.

All 7Documents 3Records 0Reporting 3People 1

1. 1 SS Samuel Salfati Paper · arxiv.org · 9 Sep 2026 Attention-Weighted Value Projection for KV-Cache Compression Preprint · may not have been peer reviewed See the source
2. 2 A arxiv.org Paper · arxiv.org Attention-Weighted Value Projection for KV-Cache Compression Preprint · may not have been peer reviewed See the source
3. 3 S Samuel Contributor · interview 10 Sep 2026 · in the article since 18 Sep 2026 Contribution — Samuel 11 statements added to this article
4. 4 A arXiv Third-party reporting · alphaxiv.org · 8 Nov 2024 Eigen Attention: Attention in Low-Rank Space for KV Cache Compression See the source
5. 5 A arXiv Third-party reporting · alphaxiv.org · 3 Mar 2025 WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models See the source
6. 6 A ar5iv.labs.arxiv.org Paper · ar5iv.labs.arxiv.org ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration Preprint · may not have been peer reviewed See the source
7. 7 A arXiv Third-party reporting · alphaxiv.org · 7 Jun 2026 STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control See the source

Also consulted: 1 page across 1 site, none cited.

Article history

1. Published 18 Sep 2026, 18:23
   Assembled by the Primary desk from 6 sources · 1 contributor · 24 cited sentences
2. S Samuel, independent researcher 11 statements added 10 Sep 2026 9 days before publication

   1. Go to the sentence: In his constructed scenario, one direction carried nine times more squared energy than the other across two input tokens, with values of 18 and two respectively.↑
   2. Go to the sentence: However, when an unmasked uniform attention operation averaged the two tokens together, the first direction vanished entirely to zero in the output, while the smaller second direction survived with an energy of two.↑
   3. Go to the sentence: Under rank-one compression, conventional uncentered principal component analysis preserved the first direction because it contained most of the isolated value energy, incurring a squared output error of two.↑
   4. Go to the sentence: In contrast, projecting onto the subspace derived from the attention-weighted output retained the second coordinate, reproducing the mixed output with zero squared error.↑
   5. Go to the sentence: He explained that "the direction carrying 9/10th of the energy cancels out to exactly zero once you average the two tokens, while the direction that looks negligible in isolation is the only thing that survives the output."↑
   6. Go to the sentence: In Salfati's paper, the author formulated the objective as minimizing the squared Frobenius norm between the true attention output and the output formed by projecting values through a low-rank orthogonal matrix.↑
   7. Go to the sentence: For a sequence with value matrix V and fixed attention weight matrix alpha, the optimal projection matrix is formed by the leading eigenvectors of the product of the transpose of V, the transpose of alpha, alpha, and V.↑
   8. Go to the sentence: When multiple query heads read from the same value states, their individual Gram matrices must be summed together with non-negative weights before performing the eigendecomposition.↑
   9. Go to the sentence: Salfati noted that replacing this weighted sum with the Gram matrix of the mean attention across heads introduces cross-head interaction terms that alter the mathematical optimizer, making the sum of individual Gram matrices the mathematically consistent choice.↑
   10. Go to the sentence: Furthermore, compressing values in earlier layers alters the queries and keys computed in subsequent layers, an interaction that independent layer-by-layer projections cannot globally optimize.↑
   11. Go to the sentence: However, Salfati clarified that these historical experiments altered static model weights and immediately reconstructed dense floating-point numbers during execution, rather than storing compact integer codes or low-rank vectors in a physical key-value cache.↑