Language model watermarks degrade unevenly across different linguistic families

Hidden statistical patterns used to identify machine-generated text break down along grammatical lines, according to a new evaluation framework tested across 11 languages.

When a large language model generates a paragraph, hidden mathematical patterns can be tucked into the choice of words to prove a machine wrote it. These watermarks are designed to remain invisible to human readers while remaining easy for automated checkers to spot. Across the technology sector, developers routinely assume that a watermark working reliably on English will behave identically when the same model translates a document or writes an essay in Turkish, Hindi, or Korean. In practice, the underlying structure of different languages changes how watermarking algorithms alter generated text and whether automated detectors can find the signal.

The issue matters because watermarking is increasingly relied upon to detect automated disinformation, enforce academic integrity rules, and track synthetic content across digital networks. If watermarking methods degrade text quality or fail to detect machine-generated writing in some languages more than others, users and developers working in non-English languages face unequal penalties and lower-quality outputs. A watermarking system that works smoothly in one language could distort grammar in another or let machine-generated text pass undetected through automated safety filters.

To understand why language properties disrupt watermarking, consider what has to happen during text generation. A language model produces text by predicting a list of probable next word pieces, known as tokens. A watermarking algorithm splits these candidate tokens into favored lists and restricted lists, gently nudging the model to select tokens from the favored list. When a detector checks the text later, it counts how often tokens from the favored list appear relative to random chance. If the language has complex word-building rules, richer morphology, or different vocabulary distributions, the available token choices change drastically. That shift alters both the statistical strength of the watermark and the fluency of the resulting sentences.

Why do standard watermark evaluations miss multilingual disparities?

Standard benchmarks evaluate watermarks almost exclusively on English text using fixed detection thresholds that obscure how performance changes across languages. In a preprint posted to arXiv on August 22, 2026, Alexander Nemecek, Osama Zafar, Debargha Ganguly, Vikash Singh, Vipin Chaudhary, and Erman Ayday report that choices in evaluation design that seem harmless on English end up dictating conclusions when applied across multiple languages.¹

The authors found that performance gaps in watermarking are structural to language properties rather than quirks of individual languages.¹ By separating language differences into distinct grammatical and historical families, the researchers observed that disparity in both detection accuracy and text quality occurred predominantly between language families rather than between specific languages within the same family.¹ Languages that share fundamental structural characteristics, such as how they inflect words or structure sentences, tend to experience similar watermarking failures.

The underlying cause lies in how tokenizers divide text into mathematical units before generation starts. Alexander Nemecek, the lead author, said in response to questions from Primary that "most watermarks operate token by token, so the language structure enters through the tokenizer of the language model."^contributed When a language forms complex words by chaining prefixes and suffixes, the tokenizer breaks words down into smaller fragments. Nemecek said that "morphologically richer languages would fragment into more, smaller tokens, which would then change the per-token statistics the watermark biases and the detector later counts."^contributed In their experiments, this fragmentation was the strongest predictor of quality loss for schemes that alter token probabilities directly, while detection failures often tracked whether the model stayed in the target language at all.^contributed

How did the researchers audit cross-lingual watermarking?

Alexander Nemecek and colleagues designed an auditing framework built around four components to measure where and why watermarking fails across different tongues.¹ First, the framework calculates detection thresholds calibrated empirically for each specific deployment context, ensuring that a checker is tuned to the baseline behavior of the language being evaluated.¹ Second, it adds a measurement independent of specific thresholds to separate calibration mistakes from genuine detection failures.¹ Third, it evaluates text quality across three distinct measurement paradigms, tracking overall word distributions, paired semantic meaning, and perplexity relative to reference text.¹ Fourth, it applies an entropy decomposition technique to measure how much disparity arises between different language families.¹

The researchers applied this framework across six distinct watermarking schemes and three open-weight language model generators.¹ Their evaluation spanned 11 languages covering four scripts and eight typological families, testing both raw base models and models tuned to follow instructions.¹ Testing across this matrix revealed that theoretical design guarantees often break down in practice. Nemecek noted that "the two watermarking schemes that are designed to be distortion-free, meaning they leave the generation distribution unchanged in expectation, turned out to be the most distorting ones empirically under all three of our quality measurements."^contributed

What can these findings not determine?

The preprint describes an evaluation framework tested on 11 languages and six watermarking schemes, which represents an extensive audit but does not cover every spoken tongue or proprietary model.¹ The reported disparities reflect the behavior of the evaluated open-weight models and watermarking algorithms under specific experimental settings, and they cannot predict the performance of proprietary commercial systems whose watermarking mechanisms remain secret. The study analyzes structural differences across language families rather than surveying every dialect or local variation within those families. Furthermore, the preprint was posted to the arXiv server and has not yet completed formal peer review.¹

The authors also caution against overinterpreting the failure rates. Nemecek said that "a step too far would be concluding that watermarking does not work outside of the English language," adding that no single method emerged as universally superior because scheme rankings shift depending on which evaluation metrics and prompts are chosen.^contributed In addition, six of the eight typological families in the study contained only one language, meaning further testing with related languages is needed to confirm family-wide patterns.^contributed

What happens next for multilingual text watermarking?

Developers building watermarking systems must recalibrate their detection tools and quality checks for the specific structural traits of each language family they support. When companies deploy a single global detection threshold or assess output quality purely against English benchmarks, they risk applying unfair standards to non-English text. Future research will need to test whether watermark algorithms can be dynamically adapted to different morphological systems without sacrificing detection reliability or output quality.

This piece was prepared from the arXiv preprint and public records together with answers from Alexander Nemecek to five questions from the Primary news team, completed August 2026.

The record

18 sentences trace to 1 source and 1 contributor.

All 2Documents 1Records 0Reporting 0People 1

1. 1 AN Alexander Nemecek, Osama Zafar, Debargha Ganguly, Vikash Singh, Vipin Chaudhary, Erman Ayday Paper · arxiv.org · 22 Aug 2026 Auditing Cross-Lingual Fairness in Language Model Watermarking Preprint · may not have been peer reviewed See the source
2. 2 AN Alexander Nemecek Contributor · interview 27 Aug 2026 · in the article since 28 Aug 2026 Contribution — Alexander Nemecek 12 statements added to this article

Article history

1. Published 28 Aug 2026, 10:20
   Assembled by the Primary desk from 1 source · 1 contributor · 18 cited sentences
2. AN Alexander Nemecek, the lead author 12 statements added 27 Aug 2026 the day before publication

   1. Go to the sentence: In a preprint posted to arXiv on August 22, 2026, Alexander Nemecek, Osama Zafar, Debargha Ganguly, Vikash Singh, Vipin Chaudhary, and Erman Ayday report that choices in evaluation design that seem harmless on English end up dictating conclusions when applied across multiple languages.↑
   2. Go to the sentence: The authors found that performance gaps in watermarking are structural to language properties rather than quirks of individual languages.↑
   3. Go to the sentence: By separating language differences into distinct grammatical and historical families, the researchers observed that disparity in both detection accuracy and text quality occurred predominantly between language families rather than between specific languages within the same family.↑
   4. Go to the sentence: Alexander Nemecek, the lead author, said in response to questions from Primary that "most watermarks operate token by token, so the language structure enters through the tokenizer of the language model."↑
   5. Go to the sentence: Nemecek said that "morphologically richer languages would fragment into more, smaller tokens, which would then change the per-token statistics the watermark biases and the detector later counts."↑
   6. Go to the sentence: In their experiments, this fragmentation was the strongest predictor of quality loss for schemes that alter token probabilities directly, while detection failures often tracked whether the model stayed in the target language at all.↑
   7. Go to the sentence: Alexander Nemecek and colleagues designed an auditing framework built around four components to measure where and why watermarking fails across different tongues.↑
   8. Go to the sentence: The researchers applied this framework across six distinct watermarking schemes and three open-weight language model generators.↑
   9. Go to the sentence: Their evaluation spanned 11 languages covering four scripts and eight typological families, testing both raw base models and models tuned to follow instructions.↑
   10. Go to the sentence: Nemecek noted that "the two watermarking schemes that are designed to be distortion-free, meaning they leave the generation distribution unchanged in expectation, turned out to be the most distorting ones empirically under all three of our quality measurements."↑
   11. Go to the sentence: Nemecek said that "a step too far would be concluding that watermarking does not work outside of the English language," adding that no single method emerged as universally superior because scheme rankings shift depending on which evaluation metrics and prompts are chosen.↑
   12. Go to the sentence: In addition, six of the eight typological families in the study contained only one language, meaning further testing with related languages is needed to confirm family-wide patterns.↑