AI safety has a friendly-fire problem. The technology that proves where AI text came from is weakening the technology that checks whether it is true. Watermarking, or embedding a hidden signature in generated text so that its origin can be proven later, is used to increase transparency on AI applications (Europe’s EU AI Act Article 50 [1] ; China’s AI labeling regulation by the Cyberspace Administration of China [2]). Besides, hallucination detection tries to flag the parts of an answer that the model made up. We use it to check if model outputs are reasonably predictable. Regulators are pushing both edges. But there is a hidden collision inside AI models between both transparency and liability. The collision is hidden in the model’s internals but can have real consequences in AI systems’ safety.
How watermarking works
An LLM does not write sentences. It computes, at every step, a probability distribution over the next token. Sometimes that distribution is sharp. After “The capital of France is”, the token “Paris” carries almost all the probability mass, and there is no real choice to make. Sometimes it is flat. After “She left the party because”, multiple continuations are equally plausible, and the model flips a weighted coin. The technical name for the flatness of that distribution is entropy or semantic entropy to be more specific [1]. High entropy means the model is choosing among many comparable options. Low entropy means the next token is essentially forced. Keep this distinction in mind, because both technologies in this story live at the high-entropy positions.
A text watermark biases the coin flips using a secret key. The best known scheme for watermarking comes from the work of Kirchenbauer and colleagues in 2023 [4]. This approach splits the vocabulary at each step into a pseudorandom “green” and “red” list derived from the key and the recent context, then adds a small bonus to the green tokens (Figure 1). The text still reads naturally, but it contains statistically more green tokens than chance would produce. A detector holding the key reconstructs the lists, counts the green tokens, and computes how improbable that count would be in unwatermarked text.
Google DeepMind’s SynthID-Text [5], deployed in all its Gemini family models, refines the idea with a mechanism called tournament sampling (the model drafts several candidate words and the secret key referees a knockout between them; the champion gets written), applied as a processor on the model’s output probabilities. The scheme is open sourced in the Hugging Face Transformers library.
A third family, the distortion-free schemes inspired by Aaronson’s Gumbel trick, arranges the bias so that the output distribution of a single generation is provably unchanged on average, while the text remains correlated with the key [6].
A watermark can only push where there is room to push. At a low-entropy position, biasing the choice would mean writing the wrong word, and quality would visibly collapse. Take an invoice summary. In the sentence “The total due is 4,320 dollars,” everything after “is” is constrained. There is only one correct continuation, and a watermark that nudged the model toward any other number would not be hiding a signature, it would be corrupting the document. Now take the sentence “Payment was received immediately” The model could have written “quickly”, “swiftly”, or “without delay”, and the reader could never say which word it would have picked on its own. That second slot is where the watermark lives. A signature can only hide inside choices that make no difference, which is exactly why it is invisible, and exactly why it cannot exist where the choice matters. So every scheme concentrates its signal at high-entropy positions, the moments of doubt. This signature is statistical and average-case. Detection is reliable “given enough text”, which is not the same as a per-output certificate.
Currently Google marks all their Gemini family models. OpenAI built a scheme and chose not to deploy it. Anthropic has just announced they are going to add watermarks to their Claude family models. Open-weight models leave decoding in the user’s hands, so their text is unmarked whenever the user prefers.
How hallucination detectors work
A hallucination is a fluent statement that happens to be wrong: an invented statistic, a fabricated citation, a number that appears in no source document. Current detectors can be divided into two mechanistically different families.
-
Grounding-based detectors read the answer against a trusted source and ask whether each claim is supported. Token-support models such as LettuceDetect classify which spans of the answer lack backing in the context. Entailment models such as MiniCheck ask whether the source logically implies the claim. LLM judges do the same with a prompted model. What is common in these approaches is the input: the output text and the evidence, nothing else.
-
Uncertainty-based detectors exploit a behavioral signature instead: a model that knows the answer produces it consistently, while a model that is guessing drifts across resamples. Semantic entropy[1] formalizes this by sampling several answers, clustering them by meaning, and measuring the entropy over clusters. SelfCheckGPT and the logprob-calibration methods are variations on the the same idea. In this family of detectors also the input is a common feature. the model’s uncertainty, read either from repeated sampling or from the probabilities directly.
The collision
The watermark operates at high-entropy positions because those are the only positions where the choice can be steered invisibly. Uncertainty-based detectors reads those positions because hesitation is its “here” ignal. The watermark steps in at the moment of doubt, and resolves it with the secret key (a private number that fixes how every doubtful choice tips). The detector’s trick depends on the resamples being fresh, independent tries. The watermark schema breaks that. Every retry uses the same secret key, so the coin flips inside the model land the same biased way each time, and in the strictest watermark schemes a fixed key produces exactly the same answer, word for word (Figure 2). If we ask a guessing model the same question five times we should see five slightly different guesses. That variation is the warning sign. With the watermark the five tries agree, not because the model stopped guessing, but because the same loaded coin decided all five. The steadiness we observe belongs to the sampler, not to the model’s knowledge.
As a consequence, consistency-based hallucination detectors overestimate agreement and underestimate uncertainty on watermarked text.
Not every confidence check works by asking the question again. A second family skips the retries and reads the model’s internal probability numbers directly, the running record of how sure it was about each word. But the watermark has already edited those numbers before any check gets to read them, so this family inherits the distortion as well. A 2025 study across seven instruction-tuned models [7] found that watermarking measurably changes how models behave on downstream tasks, and that the changes remain after accounting for the loss in text quality. So, for any check that reads retries or probability numbers, the interference follows necessarily from how the watermark works.
Grounding-based detectors sit on the other side. They read only the output text against the evidence (for example in RAG systems), and the watermark selects among semantically equivalent continuations. So, in principle, they are not affected by watermarks as much as operate in a different space. If we need to run reliability checks on watermarked text, this is the family to run.
The watermark removers
The removal tools arrived on schedule, and nearly all of them are paraphrasers: a second model reads the watermarked text and says it again in different words. We are currently seing three approaches:
-
Watermark stealing [8] showed that for under fifty dollars of API queries an attacker can learn enough of a scheme’s hidden green-list rule to strip the mark from schemes previously considered safe, with success rates above 80 percent, and also to forge it.
-
BIRA [9] (September 2025) rewrites text while steering the rewriter away from the words the watermark favored, and reports over 99 percent evasion without knowing which scheme was used.
-
WASH [10] (May 2026) does not even rewrite: it averages the outputs of three ordinary models, the independent watermark biases cancel out, and detection scores fall from far above the alarm threshold to below it.
The question is, do they work? The response is yes, and this is settled and it has a theoretical explanation. There is a mathematical proof of this with a fitting title: “Watermarks in the sand: Impossibility of strong watermarking for generative models” [11]. The argument is simple (Figure 4). Suppose the attacker can do two things: tell whether a small edit made the text worse, and keep making small edits that don’t. Then they can wander step by step through thousands of rewrites that are all equally good, and somewhere along that walk the watermark’s pattern gets left behind, because the pattern lived in the specific word choices and the words are no longer the ones the model chose. No current watermark scheme can survive that, whatever key it uses. Google, in its own description of SynthID-Text presents it as a marker for good-faith use, not as protection against someone determined to remove it.
Squeezing a balloon
Remember which detectors the watermark could not hurt: the ones that never ask how confident the model felt. They take the output answer and hold it up against your source documents, checking word by word and phrase by phrase whether each claim is grounded in the provided context. The watermark changed which words got picked, but the claims still matched the sources, so these checks kept working. But the washing watermark-removal approach breaks exactly that safety. A washed text says the same thing in different words. Different words no longer line up with the source documents, so the matching scores fall, and a correct answer starts to look unsupported.
Moreover, the washing tool is itself an “AI rewriting the text”, and nobody is checking its output. Rewriting is known to change things at the edges: a number shifts, a “probably” disappears, a name gets swapped. The cleanup step can create the very mistakes the safety checks were built to catch. And since washing happens after the text is finished, it cannot bring back the natural variation the watermark removed. Conclusion: the confidence-based checks stay broken.
The distortion only moves
The distortion produced by the watermark and its removal process never disappears. It moves, like squeezing a balloon. The watermark presses on one side at writing time and distorts the checks that read doubt. The remover presses on the other side at cleanup time and distorts the checks that read evidence. Text that has been through both steps has weakened both kinds of hallucination-detection approaches. As far as I know, nobody has measured that second effect directly yet. The experiment is sitting there waiting for someone to run it.
Forgery closes the loop from the other side. The same tricks that strip a watermark off can stamp one onto text the model never wrote. So the label “watermarked” can fail both ways: missing from machine text that was laundered, present on false text that was faked.
💬 Comments and suggestions are wellcome.
✉️ You can contact me javier@jmarin.info
References
[1] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance) [https://eur-lex.europa.eu/eli/reg/2024/1689/oj]
[2] Measures for Labeling of AI-Generated Synthetic Content Document. State Information Office Tongzi [2025] №2. Cybersecurity Administration, Ministry of Industry and Information Technology, Ministry of Public Security, State Administration of Radio and Television [https://www.cac.gov.cn/2025-03/14/c_1743654684782215.htm]
[3] Farquhar, S., Kossen, J., Kuhn, L. et al. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630 (2024). https://doi.org/10.1038/s41586-024-07421-0
[4] Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023, July). A watermark for large language models. In International conference on machine learning (pp. 17061–17084). PMLR.
[5] Google DeepMind, SynthID documentation and the Hugging Face Transformers integration (SynthIDTextWatermarkLogitsProcessor, BayesianDetectorModel). https://deepmind.google/technologies/synthid/ and https://huggingface.co/docs/transformers
[6] Fu, J., Zhao, X., Yang, R., Zhang, Y., Chen, J., & Xiao, Y. (2024, August). Gumbelsoft: Diversified language model watermarking via the gumbelmax-trick. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5791–5808).
[7] Verma, A., Phan, N., & Trivedi, S. (2025). Watermarking degrades alignment in language models: Analysis and mitigation. arXiv preprint arXiv:2506.04462.
[8] Jovanović, N., Staab, R., & Vechev, M. (2024). Watermark stealing in large language models. arXiv preprint arXiv:2402.19361.
[9] Hwang, J., Park, S., & Ok, J. (2025). LLM Watermark Evasion via Bias Inversion. arXiv preprint arXiv:2509.23019.
[10] Wu, Z., Gong, G., Zhu, Q., Chen, Y., & Zhao, R. (2026). Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs. arXiv preprint arXiv:2605.30501.
[11] Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023). Watermarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.04378.