OpenAI's textGrain watermark weakens after edits and on short text, clouding E.U. compliance

The European Union now requires AI-generated text to be watermarked so it can be identified as such. IEEE Spectrum reports that tests of OpenAI's new textGrain watermark, released in October and developed to comply with the E.U. law, show the marks often do not survive additional editing or texts under 150 words. Because there is no clear legal benchmark for how well a watermark must work, it is uncertain whether textGrain will meet Europe's transparency rules.

How it works: a language model picks each next token based on the text so far. A watermark uses a secret key to generate pseudorandom values that influence that choice, leaving a statistical pattern a detector holding the same key can look for. What sets textGrain apart, according to Xiang Li, coauthor of the textGrain technical report and a postdoctoral researcher at the University of Pennsylvania, is that it treats entropy (a measure of how unpredictable the model's word choices are) as a budgetable resource. Developers specify how much unpredictability they will allow, and textGrain adjusts to that target. Li says: "We can think of entropy as a resource, a kind of money." The mark is invisible to readers. A positive result does not show how much a person contributed to a passage, and a negative result does not prove a text was human-written.

The numbers: OpenAI's false-positive rate is 1 percent, meaning the detector is designed to wrongly flag about one in 100 unwatermarked passages. In tests on psychology questions from the ELI5 database (questions and answers from the r/explainlikeimfive subreddit), watermarks were detected in about 80 percent of 200-token text (about 150 words) against 95 percent of 400-token text (about 300 words). Li calls this a "basic limitation" of text watermarking that would be complicated to solve: short passages may not hold enough watermark evidence for reliable detection, even with additional computation, while longer ones generally hold more.

Editing hurts too. In an evaluation of 400-token passages, replacing just 10 percent of words with synonyms cut detection from 92 percent to 66 percent, and replacing up to 25 percent cut it to 17 percent, according to OpenAI. Because the watermark depends on preceding tokens, Li says, changing one word can disrupt the signal at several positions. OpenAI says watermarking alone is not enough to decide whether content is AI-generated, and that textGrain will "complement" other provenance tools, such as content credentials in metadata and Google DeepMind's SynthID watermarks for images and audio. OpenAI did not respond to a request for comment.

The legal side: the transparency obligations in Article 50 of the E.U. AI Act went into effect on 2 August. They require providers of systems that generate synthetic audio, images, video or text to mark outputs in a machine-readable format so they can be detected as artificially generated or manipulated. Technical solutions must be "effective, interoperable, robust and reliable as far as this is technically feasible." The E.U.'s Code of Practice offers a voluntary way to show compliance, and it does not require watermarks on outputs shorter than 200 tokens. Thomas Souverain, a postdoctoral fellow on AI Ethics at the Institute of Advanced Education in Paris, says the legislation is open to interpretation.

Souverain says textGrain's early methodology seems reliable and could beat competitors at identifying AI text. OpenAI says textGrain matched or outperformed Google's SynthID Text in its own tests, but those results have not been independently verified. Until independent researchers can test it, and while the AI Act's criteria lack measurable thresholds, it is too early to say whether textGrain meets legal requirements. Souverain notes there is only a technical study, no peer-reviewed article, and no access yet to a precise protocol. If textGrain is eventually found not to meet the requirements, he says, OpenAI could face a fine of up to 7 percent of its annual income.

OpenAI is working with a limited group of researchers, including John Thickstun, an assistant professor of computer science at Cornell University, and a team at the Kempelen Institute of Intelligent Technologies (KInIT) in Bratislava, Slovakia. Thickstun and Dominik Macko, a senior researcher at KInIT, say they had only just gained access and had not begun evaluating the system. Macko says his team plans to test robustness across subject areas, languages and text lengths. Souverain calls on OpenAI to make textGrain open-source so outside researchers can try to break it, saying he would like an even more open phase, as with SynthID, where the methods are accessible.

Key facts

OpenAI's textGrain text watermark, released in October to comply with the E.U. AI Act, often does not survive editing or texts under 150 words, per IEEE Spectrum.
Detection was about 80 percent on 200-token text (about 150 words) and 95 percent on 400-token text (about 300 words), with a 1 percent false-positive rate.
On 400-token passages, replacing 10 percent of words with synonyms cut detection from 92 percent to 66 percent; up to 25 percent cut it to 17 percent, according to OpenAI.
The AI Act's Article 50 has no measurable threshold for watermark quality, so it is too early to say whether textGrain meets it; Souverain says a fine of up to 7 percent of annual income is possible if it does not.
OpenAI's claim that textGrain matches or beats Google's SynthID Text is unverified; outside testers at Cornell and KInIT have only just got access.

Why it matters

Article 50 of the E.U. AI Act has been in effect since 2 August and requires machine-readable marking of AI-generated text, among other outputs. textGrain is OpenAI's answer, and it is uncertain how well a text watermark holds up against vague legal wording. Its novelty is treating entropy as a budget, giving developers control over the trade-off between a detectable signal and variety in the model's output. The reported weak spots, short and edited text, are exactly where real-world use is likely to be messy.

Who it affects

OpenAI, which faces a possible fine of up to 7 percent of annual income if textGrain is found not to meet the E.U. requirements, according to Souverain. Other providers of text-generating systems face the same Article 50 duty and the same vague standard. Readers, publishers and anyone trying to detect AI text are affected too: a positive result does not show how much a person contributed, and a negative one does not prove human authorship.

How to use it

textGrain is not open to the public. OpenAI is working with a limited group of researchers, including John Thickstun at Cornell and Dominik Macko's team at KInIT. For anyone building on or assessing watermarking, the practical takeaways from the article are to treat detection as probabilistic, expect weaker results below about 150 words, and pair watermarks with other provenance tools such as content credentials in metadata, as OpenAI itself suggests.

How solid is it

The detection figures come from OpenAI's evaluation, and the editing figures are explicitly attributed to OpenAI. OpenAI's claim that textGrain matched or outperformed SynthID Text in its own tests has not been independently verified. Souverain points out that there is only a technical study, with no peer-reviewed article and no precise protocol available yet. The article itself does not say textGrain fails the E.U. rules; it says it is too early to tell. OpenAI did not respond to a request for comment.

Risks and caveats

The legal question is open: the AI Act's criteria lack measurable thresholds, and Souverain says the law is open to interpretation. The E.U.'s voluntary Code of Practice does not require watermarks on outputs shorter than 200 tokens. The 7 percent fine is a conditional possibility stated by Souverain, not a finding. Test coverage so far is narrow: the reported results use psychology questions from ELI5, and Macko's team plans to test other subject areas, languages and text lengths. Souverain also wants textGrain open-sourced so outsiders can look for faults, paraphrase attacks and translation weaknesses.

“We can think of entropy as a resource, a kind of money.”

— Xiang Li, coauthor of the textGrain technical report and postdoctoral researcher at the University of Pennsylvania