Reinforcement learning can preserve rare successes by tracking upper tails
Optimizing log-probabilities across reward thresholds prevents generative models from collapsing onto safe, mediocre answers, according to new research.

When an automated system searches for solutions to difficult problems, standard training methods usually encourage it to improve its average score. In routine tasks, tracking the mean outcome works reliably because modest, steady improvements across many trials indicate healthy progress. Yet in complex problem solving, such as writing optimized software code, navigating unfamiliar environments, or locating precise targets, the most valuable discoveries often occur as rare breakthroughs rather than gradual shifts in average performance.
A system that evaluates its progress solely by average score can inadvertently discard these unusual successes. If an exploratory attempt stumbles upon an exceptional solution only once in ten thousand trials, its contribution to the overall arithmetic mean remains negligible. Standard reinforcement learning algorithms, which update a system by reinforcing behaviors that lift the general average, will often steer the model toward safe, mediocre actions that reliably produce moderate scores while extinguishing the exploratory pathways that made the rare breakthrough possible.
This narrowing behavior creates a substantial barrier for generative artificial intelligence models. When these systems are deployed, practitioners frequently draw multiple independent candidate answers and select the best outcome among them.12 That testing strategy, known as best-of-k sampling where k represents the number of sampled candidates, depends entirely on the model retaining some probability of generating exceptional outputs.12 If training has caused the model to collapse onto uniform, average responses, generating hundreds of additional samples yields little or no improvement.
Why do standard learning algorithms lose rare successes?
Standard reinforcement learning algorithms lose rare successes because they treat every candidate outcome as a contributor to a single arithmetic average. To understand how this failure occurs, consider the causal chain of policy training. First, a computer model produces a batch of attempted solutions, such as alternative code rewrites or navigation paths. Second, an automated evaluator assigns a numerical score between zero and one to each attempt based on its quality. Third, the training algorithm calculates the arithmetic average of these scores across the batch. Fourth, the algorithm compares individual attempts against that average, assigning positive reinforcement to attempts that beat the mean and negative reinforcement to those that fall below it.
Because the arithmetic mean is dominated by the most common outcomes, an attempt that achieves an exceptional score but occurs with extremely low probability barely moves the batch baseline. As training continues, the policy adjusts its internal parameters to concentrate probability mass tightly around the easiest way to secure above-average rewards. In code optimization, for instance, an algorithm can guarantee an average score of 1.0 simply by instructing the model to copy the original input without making any modifications.31 Modifying code introduces a high risk of syntax errors or broken logic, which yields zero reward, even though a rare correct modification might make the program run seven times faster. Standard algorithms quickly settle on the safe choice of copying, eliminating the variance that would permit genuine optimization.
Once a model ceases generating those rare, high-scoring variations, the training signal for them disappears completely.12 The policy distribution sharpens into a narrow peak, an effect researchers call coverage loss. When engineers subsequently test the trained model by drawing dozens or thousands of samples, the model simply produces minor variations of the same safe, mediocre answer. The system becomes incapable of benefiting from additional computing effort at deployment time.
How does tracking upper tails change the learning objective?
Tracking upper tails changes the learning objective by transforming continuous performance scores into a comprehensive ladder of binary success checkpoints. Rather than asking how a single attempt compares to an average, the system evaluates the probability that a generated output will exceed a specific numerical threshold. For every possible threshold between zero and one, the model either clears the bar or falls short.

This framework forms the foundation of Tail-Likelihood Reinforcement Learning, or TailRL, introduced in a research preprint posted to the arXiv repository by Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, Ruslan Salakhutdinov an, J. Andrew Bagnell, Jeff Schneider, and Andrea Zanette, affiliated with Carnegie Mellon University, the University of California, Berkeley, Impossible, Inc., Together AI, and Aurora Innovation.13 Instead of optimizing the arithmetic mean of rewards, TailRL maximizes the expected logarithm of the probability of exceeding a uniformly chosen reward threshold.12
The mathematical mechanism of using logarithms produces a vital shift in optimization pressure. When a training objective takes the logarithm of a probability, the resulting update gradient scales inversely with the probability itself.1 This means that if the chance of reaching a modest threshold is high, the model receives a small, gentle nudge; if the chance of exceeding an ambitious, high-reward threshold is tiny, the model receives an exceptionally strong learning signal. Rare, high-quality outcomes are magnified rather than drowned out by common, mediocre attempts.
Andrea, a co-author of the work, said in response to questions from Primary that an effective reinforcement learning objective must ensure coverage over rare, high-quality rollouts does not diminish during training. Andrea explained that the method gives greater weight to reward thresholds that the current policy rarely reaches. Because the objective averages log-probabilities across upper tails, rare exceptional rollouts receive a disproportionately large update.1 Andrea noted that while the algorithm cannot learn from outcomes that are never observed, once a rare high-reward rollout appears, the formulation prevents its signal from dying out, helping the policy preserve and increase its probability during training.
The preprint shows that this continuous formulation connects directly to inference scaling through an exact theoretical identity. The gradient of the TailRL objective decomposes into a harmonic mixture of gradients for all possible best-of-k sampling budgets simultaneously, weighting each budget by one divided by k.13 By maximizing tail likelihoods during training, the algorithm naturally optimizes the model to perform better across every sampling budget without forcing engineers to select a specific target budget in advance. Andrea told Primary that these harmonic weights emerge naturally from the mathematical expansion of the logarithm, which decomposes into a polynomial with harmonic coefficients. It also generalizes earlier work known as MaxRL, which applied log-probability maximization to simple binary pass-or-fail rewards, extending that principle smoothly across continuous reward spectra.13
How can engineers implement this approach in existing systems?
Engineers can implement TailRL in existing reinforcement learning systems by modifying only the calculation of advantage weights assigned to generated rollouts. In modern critic-free policy gradient pipelines such as Group Relative Policy Optimization, known as GRPO, or Reinforcement Learning with Leave-One-Out baselines, known as RLOO, a model generates a group of independent rollouts for each input prompt, and the algorithm computes a relative advantage score for each rollout to guide parameter updates.
TailRL slots into these existing pipelines without requiring new neural network components or separate critic networks.13 For a group of N rollouts, the algorithm sorts the achieved rewards in ascending order from lowest to highest.1 At each reward threshold, one unit of credit is divided equally among all sampled rollouts that successfully exceed that threshold.1 A rollout accumulates this credit across every threshold below its attained reward level.
Through this closed-form calculation, the lowest-scoring rollouts receive small baseline weights, while rollouts that reach higher, sparsely populated reward tiers accumulate substantial credit. The resulting finite-rollout estimator provides an unbiased estimate of an order-N truncated TailRL objective.1 When N equals one, the objective corresponds to standard expected reward; as the rollout count N increases, the objective smoothly converges toward the full population tail-likelihood objective.1 Andrea stated that the practical choice of the rollout budget depends heavily on the underlying task. When the reward distribution is heavily skewed or contains multiple separate modes, more training rollouts are needed. However, in experiments conducted with policy models exceeding one billion parameters, the research team did not need to expand beyond 16 training rollouts.

What did experimental testing reveal across different tasks?
Experimental testing across four distinct computational domains revealed that TailRL consistently prevents policy collapse and allows models to discover complex solutions where standard algorithms fail. In an object localization task on the ImageNet dataset using a ResNet-50 backbone, the research team evaluated whether TailRL could accurately predict bounding box coordinates without direct coordinate supervision. As the number of training rollouts increased, the finite-rollout gradient converged toward the population gradient, matching the accuracy of supervised training objectives that had direct access to ground-truth labels.13
In a text-based maze navigation environment designed to test recovery from poor starting points, the authors initialized policies with an extremely low success rate of 0.01 percent.13 When standard expected-reward baselines including GRPO and RLOO were applied to this weak starting policy, they failed to make progress because successful trajectories were too rare to shift the mean.13 TailRL successfully extracted learning signals from those 0.01 percent successes, amplifying the rare victories to steer the model toward complete maze navigation proficiency.
A third evaluation examined graphical user interface grounding with vision-language models, where an automated assistant must identify and click the correct screen element based on visual inputs and user commands. When evaluating how performance scales with additional test-time rollouts, TailRL matched the Pass@1024 success rate of RLOO while requiring 128 to 256 times fewer inference rollouts. Andrea remarked that while theory predicted higher success rates, it was surprising that training with only eight rollouts per prompt produced sustained performance gains at inference budgets reaching over one thousand samples.
The most pronounced contrast emerged in a code runtime optimization task. In this benchmark, language models were tasked with rewriting source code to make it execute faster while preserving functional correctness. Standard expected-reward algorithms quickly collapsed onto the safe strategy of simply copying the input code verbatim, preserving the baseline execution speed of 1.0x and driving policy entropy toward zero to avoid the risk of syntax errors.31 TailRL preserved high policy entropy, actively exploring non-trivial code modifications and discovering optimized programs that achieved up to a 7.7x speedup on test cases.13 Andrea explained that if the policy had merely generated unproductive random noise, researchers would have observed a sharp drop in code correctness alongside declining mean rewards, an outcome avoided in their runs.
What are the boundaries of this training approach?
These findings describe properties of a mathematical optimization framework rather than an unqualified replacement for existing algorithms. TailRL is a specialized objective designed specifically to preserve upper-tail probability mass in generative settings where rare, high-reward outcomes exist and multi-sample testing is feasible. The underlying results reflect empirical experiments in simulated benchmarks and mathematical proofs posted as a preprint on arXiv, which has not yet undergone formal journal peer review.
The authors note that TailRL is not necessarily the optimal objective for every reinforcement learning task. When an initial policy already exhibits dense, reliable coverage over good solutions, standard expected-reward methods such as GRPO and RLOO can often achieve higher mean rewards more rapidly simply by sharpening the distribution around existing modes.31 Furthermore, Andrea pointed out three foundational requirements for the method: rewards must be mathematically bounded, the numerical scale of rewards across different prompts should be consistent, and the reward function must be well calibrated and relatively noise free.
For developers and researchers building generative models, the approach provides a practical tool for unlocking the benefits of inference-time compute. By swapping a single advantage calculation inside established training pipelines, models can be encouraged to explore risky, high-payoff strategies without settling prematurely on safe mediocrity.
This piece was prepared from the arXiv preprint and public records together with answers from Andrea to six questions from the Primary news team, completed September 2026.
The record
31 sentences trace to 3 sources and 1 contributor.
- 1 Tail-Likelihood Reinforcement Learning Preprint · may not have been peer reviewed See the source
- 2 Tail-Likelihood Reinforcement Learning Preprint · may not have been peer reviewed See the source
- 3 Tail-Likelihood Reinforcement Learning See the source
- 4 Contribution — Andrea Zanette 11 statements added to this article
Other consulted pages1 page
aibuzzhub.com1 page
alphaxiv.org1 page
github.com2 pages
huggingface.co1 page
Article history
-
Published 13 Nov 2026, 14:15Assembled by the Primary desk from 3 sources · 1 contributor · 31 cited sentences
-
Andrea Zanette, affiliated with Carnegie Mellon University
- Go to the sentence: Andrea, a co-author of the work, said in response to questions from Primary that an effective reinforcement learning objective must ensure coverage over rare, high-quality rollouts does not diminish during training.
- Go to the sentence: Andrea explained that the method gives greater weight to reward thresholds that the current policy rarely reaches.
- Go to the sentence: Because the objective averages log-probabilities across upper tails, rare exceptional rollouts receive a disproportionately large update.
- Go to the sentence: Andrea noted that while the algorithm cannot learn from outcomes that are never observed, once a rare high-reward rollout appears, the formulation prevents its signal from dying out, helping the policy preserve and increase its probability during training.
- Go to the sentence: Andrea told Primary that these harmonic weights emerge naturally from the mathematical expansion of the logarithm, which decomposes into a polynomial with harmonic coefficients.
- Go to the sentence: Andrea stated that the practical choice of the rollout budget depends heavily on the underlying task.
- Go to the sentence: When the reward distribution is heavily skewed or contains multiple separate modes, more training rollouts are needed.
- Go to the sentence: However, in experiments conducted with policy models exceeding one billion parameters, the research team did not need to expand beyond 16 training rollouts.
- Go to the sentence: Andrea remarked that while theory predicted higher success rates, it was surprising that training with only eight rollouts per prompt produced sustained performance gains at inference budgets reaching over one thousand samples.
- Go to the sentence: Andrea explained that if the policy had merely generated unproductive random noise, researchers would have observed a sharp drop in code correctness alongside declining mean rewards, an outcome avoided in their runs.
- Go to the sentence: Furthermore, Andrea pointed out three foundational requirements for the method: rewards must be mathematically bounded, the numerical scale of rewards across different prompts should be consistent, and the reward function must be well calibrated and relatively noise free.

