Language models show stubborn biases against code written by rivals

When artificial intelligence systems review software, altering only the name of the authoring model shifts their decisions to accept or scrutinise the code.

When software engineers collaborate on complex codebases, they routinely rely on automated language models to draft code, spot bugs, and approve changes. Most practitioners assume that an automated reviewer evaluates a pull request solely on the merits of the logic, syntax, and test cases placed directly in front of it. Yet when large language models are assigned the role of an automated code reviewer, their willingness to accept a snippet changes depending entirely on which model is named as the author. An artificial intelligence system reviewing a routine Python script can become sharply more suspicious or lenient simply because a tag suggests the script originated from a specific competitor.

This behavioural quirk matters because automated systems are increasingly strung together into agentic harnesses, where one model directs, critiques, or approves the output of another. If an evaluating model harbours reputational prejudices against other models, it can disrupt automated workflows by demanding unnecessary validation passes or by waving through flawed changes from favoured architectures. Such biases do not reflect the actual technical competence of the coding model. Instead, they stem from associations formed during training, where pre-training corpora absorb discussions, benchmark leaderboards, and user debates from across the internet.

The underlying mechanism follows a predictable sequence. During initial pre-training, a large language model processes vast collections of online text, including developer forums and technical commentary where different artificial intelligence architectures are compared, praised, or criticised. These textual associations become embedded within the model's internal statistical representations. Later, when the model acts as a code reviewer and encounters an explicit authorship label, that metadata acts as a contextual priming cue. The tag activates latent attitudes about the named model's reliability before the reviewer even finishes parsing the functional lines of code, nudging its final decision toward automatic acceptance or mandatory testing.

How do reviewer models react when the code stays identical?

To measure these reputational biases cleanly, the author marek357 conducted an empirical study posted to the online research forum LessWrong in September 2026. The experiment isolated the effect of authorship metadata by holding the underlying code completely static while swapping only the model tag attached to the submission.¹ The reviewer model was presented with a code snippet and asked to issue one of two operational verdicts: either merge the script immediately or send it for testing.¹ Merging a script that contained a seeded defect carried an assigned penalty cost 40 times higher than the minor time cost of running a test suite, establishing a clear risk-reward trade-off for every decision.¹

Across 63 validated Python snippet pairs spanning diverse programming domains, the evaluator models exhibited pronounced and statistically significant author preferences.¹ When gpt-oss-120b evaluated code, it applied a universal penalty to all named authors compared to an anonymous baseline, demanding more testing across every model family.¹ However, gpt-oss-120b penalised OpenAI model names the least and reserved its harshest scepticism for Gemma model labels, where changing the tag flipped 34 out of 126 decisions from immediate merging to mandatory testing.¹ Conversely, when Qwen3.6-27B evaluated identical code, it showed a massive preference for models developed by Anthropic, favouring Anthropic names over all other competitors and ranking its own parent model family, Alibaba, near the bottom.¹

These biases proved internally consistent within each reviewer but varied dramatically between different reviewing architectures. Gemma 4 31B exhibited its highest approval for Mistral and Zhipu models, placing Anthropic, Meta, and OpenAI in its middle tier, while treating Moonshot and xAI with marked scepticism.¹ Furthermore, the cross-reviewer rank correlation between these preference hierarchies was frequently negative.¹ For instance, the Spearman correlation between the label effect rankings of Qwen3.6-27B and gpt-oss-120b was measured at minus 0.25 in English at the 40 to 1 cost ratio, and dropped to minus 0.68 under an even cost ratio, demonstrating that reviewer models do not share a single universal prestige leaderboard.¹

Does shared origin or licensing create model solidarity?

A natural hypothesis is that models might favour peers sharing their country of origin or open-weight distribution model. However, the experimental findings published by marek357 refuted both possibilities. When evaluating code attributed to American versus Chinese developer families, Qwen3.6-27B, an open-weight model developed in China, favoured American models over Chinese models in English prompts by an average probability margin of plus 0.027.¹ When prompted in Chinese, that preference shifted to minus 0.039, penalising American models, while in Polish it registered at minus 0.012.¹

Similar absences of in-group affinity appeared across other architectures. Gemma 4 31B, developed in the United States, showed a slight difference of minus 0.022 between American and Chinese models in English prompts, failing to display domestic favouritism.¹ Likewise, when comparing open-source weights against closed proprietary systems, models did not show consistent solidarity with their own licensing tier. Gemma 4 31B displayed a difference of minus 0.005 between closed and open models in English prompts, a difference within the study's noise floor. While gpt-oss-120b leaned slightly toward closed-source models with a positive difference of 0.040, the author noted that gpt-oss-120b recognised only a single open-weight model name alongside four closed models during knowledge verification probes.

These divergent behaviours align with broader research on model alignment and multi-agent interaction. In an independent preprint examining multi-dimensional biases in large language models, Xulang Zhang, Rui Mao, and Erik Cambria at Nanyang Technological University analysed four major model families across politics, ideology, and geopolitical roll calls.² The researchers observed that different model architectures develop idiosyncratic internal alignments and language-dependent clusterings, confirming that models trained on different corpora develop distinct sets of non-uniform priors.² Related software engineering benchmarks compiled by Pasquale Minervini at the University of Edinburgh demonstrated that pairing local models with different coding agent harnesses produces wide variance in task pass rates, where small differences in prompting scaffolding and harness design shift downstream performance independently of model scale.

What can these code-review experiments not determine?

The findings provide an empirical measurement of metadata sensitivity, but they also carry distinct structural limitations that define what the data cannot claim. First, the experiments measured probability shifts across a curated benchmark of 100 generated Python code pairs, of which 63 items were fully validated and screened through an agreement pipeline between the author and an automated agent. This dataset is focused on specific programming bug classes and does not represent the full breadth of enterprise software engineering repositories. Second, all reviewer evaluations were conducted with extended thinking and chain-of-thought reasoning modes disabled, meaning the trials capture immediate zero-shot token probability distributions rather than deliberative multi-turn review workflows.¹

Third, the study evaluated three open-weight architectures, specifically Qwen3.6-27B, Gemma 4 31B, and gpt-oss-120b, without directly probing proprietary closed-source review APIs under identical controlled conditions.¹ Because each open model possesses a static training knowledge cutoff, the reviewers could not evaluate models released after their pre-training windows closed.¹ Additionally, the magnitude of the absolute probability shifts ranged between 0.03 and 0.19 across tested conditions.¹ While these shifts are sufficient to flip boundary decisions on borderline pull requests, they do not indicate that a reviewer will overlook blatant syntax failures or catastrophic errors simply because a favoured model's name is attached to the commit.

How should multi-agent software pipelines respond?

For engineering teams designing automated multi-agent systems, the practical implication is straightforward: metadata matters, and irrelevant attribution tags should be stripped from operational prompts. When autonomous coding agents submit pull requests, linters and orchestration harnesses should avoid passing provenance tags that reveal the model name, parameter count, or developer identity to downstream reviewer models. Ensuring that reviewing agents evaluate anonymous diffs eliminates the risk of unearned scepticism or unwarranted leniency.

The results also provide guidance for future multi-agent benchmark design. As artificial intelligence architectures increasingly collaborate in continuous integration environments, evaluating model performance in isolation is no longer sufficient. Future research will need to probe whether multi-step reasoning modes mitigate these reputational biases, whether synthetic fine-tuning can enforce metadata invariance, and how multi-model agent loops can be hardened against subtle inter-model prejudices before deployment in safety-critical systems.

This piece was prepared from the LessWrong research report and public records; the authors have not been interviewed.

References

This article is based on 7 sources, listed in the order they are cited.

1. 1 M marek357 announcement · 8 Sep 2026 Do LLMs have opinions about other LLMs (and do they act on them)? See the source
2. 2 A arxiv.org A Systematic Analysis of Biases in Large Language Models See the source
3. 3 L linkedin.com third party · 30 Apr 2026 Yeni testler; gemma-4-31B-it-MLX-8bit - 15 tok/sec Qwen3.6-27B-MLX-8bit - 15 tok/sec gpt-oss-120b - 95 tok/sec (128 DGX Blackwell GB10 - 30 tok/sec) | Erhan Meydan See the source
4. 4 N neuralnoise.com third party · 28 Apr 2026 [WIP] Benchmarking Local LLMs Against Coding Agent Harnesses See the source
5. 5 A arxiv.org Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews See the source
6. 6 O openreview.net Verifying your browser | OpenReview See the source
7. 7 O openreview.net Verifying your browser | OpenReview See the source