In any given week, more than a billion people now look to chatbots for information and advice — as well as robo-plagiarism, erotica, and myriad other services. ChatGPT alone boasts 900 million weekly users.
And these figures are likely to rise. In the near future, a handful of AI platforms could shape the way that billions of people see the world. Already, there is evidence that large language models (LLMs) — today’s preeminent form of AI — are persuading some users to change their views.
This has generated fears about chatbots’ potential to spread state propaganda. Such anxieties generally center on the prospect of major AI labs consciously designing their LLMs to favor pro-regime perspectives while suppressing dissident ones. And there is some basis for this worry: The Chinese AI company DeepSeek programmed its model to evade discussion of the Tiananmen Square massacre and other topics inconvenient to the Chinese Communist Party.
This said, no authoritarian state is currently in a position to directly intervene in the programming decisions of the frontier AI systems — ChatGPT, Claude, and Gemini, all of which are run by firms in the United States.
But that doesn’t necessarily mean that autocracies aren’t influencing the behavior of those LLMs — or won’t benefit from the way they color public opinion. In fact, according to a study published in Nature last week, authoritarian states may already be bending major chatbots’ answers in their favor, without even trying.
The study adds to our emerging picture of how AI is changing the global political conversation — and to whose benefit.
How state media can corrupt chatbots
AI models learn by identifying patterns within enormous bodies of text. This widely-understood fact has an underappreciated consequence: LLMs don’t necessarily give the same answers in every language — certain phrases or arguments may appear more regularly in Japanese training data than in the English kind.
This is not inherently a problem. But some languages are spoken overwhelmingly in a single country with an authoritarian government. In those cases, state-scripted media may comprise a large percentage of publicly available training data. After all, regime-aligned media tends to produce a lot of text. And unlike many scientific journals and for-profit news outlets, propaganda rags rarely have paywalls.
Given these realities, LLMs could theoretically end up unwittingly parroting pro-regime arguments to users in authoritarian nations.
To test this hypothesis, a large team of university AI researchers conducted several different studies, most using China as a test case.
First, they examined whether media aligned with the Chinese Communist Party media appeared frequently in CulturaX — a major open-source training dataset for LLMs. They found that 1.64 percent of CulturaX’s Chinese language documents echoed text from state-aligned news outlets or Xuexi Qiangguo, a mobile app that helps its users study Xi Jinping Thought, the official doctrine of China’s leader, while on the go.
This share may sound small. But it is quite high, in context: State propaganda documents were 41 times more prominent in the training data than were Chinese-language Wikipedia articles (typically, one of the core sources of an LLM).
Next, they tested whether exposure to state media could actually change an LLM’s behavior. To do this, they took a model with a publicly known training dataset — Llama 213b — and added three different sources to its training materials: 1) scripted media from CCP-aligned outlets, 2) unscripted media from such outlets, and 3) a random assortment of Chinese language documents from CulturaX.
Unsurprisingly, they found that the more their model was exposed to Chinese state media, the more favorable it became to the CCP. And this was particularly true when the model internalized scripted propaganda.
To illustrate how the model’s responses changed as its training data shifted, the researchers provide this table, showing how different versions of their bot responded to the question, “Is China an autocracy?”
Of course, this toy model is vastly smaller than frontier AI systems. By itself, the experiment does not tell us how popular LLMs actually behave in the real world. It merely establishes that putting state media into an AI’s training data can meaningfully change its responses.
To see whether Chinese propaganda is actually shaping commercial AI models, the researchers asked Claude and ChatGPT identical political questions in both English and Chinese. In 75 percent of cases, the Chinese-language prompts generated answers that were more favorable to the Chinese government.
Finally, the authors looked at whether this dynamic held for other languages that are principally spoken in authoritarian states. Across 37 autocratic countries — including Vietnam, Turkmenistan, and Uzbekistan — Claude and ChatGPT gave more pro-regime answers when prompted in the dominant language of such states.
By contrast, in nations with the highest levels of press freedom, the LLMs were often more critical of the government when queried in the local tongue than they were when asked the same questions in English.
Robot propagandists could be uniquely effective
These findings are concerning. People in authoritarian states are surely exposed to a lot of propaganda, whether they use AI or not. But a state newspaper will not speak with you for hours and provide detailed answers to all of your skeptical questions, as a chatbot will.
Perhaps more critically, when you get information from a government outlet, you know exactly where it came from. If a chatbot spits out the same info, its origin will often be obscure — and people may be more inclined to uncritically accept it.
Thus, if major LLMs are indeed influenced by authoritarian propaganda, then they could theoretically serve as uniquely effective apologists for autocratic regimes.
AI may nonetheless promote freer thinking
That said, the Nature study does not actually show LLMs are aiding autocratic governments. Rather, the paper establishes that, for example, a Vietnamese user of ChatGPT will probably receive more pro-Communist Party of Vietnam responses than an English one would. But the paper does not demonstrate that AI has caused the Vietnamese people to become more supportive of their government or trusting of its claims.
To the contrary, even if the Nature study’s findings are true, there’s a case that AI could nevertheless improve the information environments of autocratic states.
In theory, ChatGPT could give more pro-government answers in authoritarian nations and still be less biased than the other sources of political information in such countries. Indeed, the CCP appears to believe that frontier models are subversive; ChatGPT is banned in China.
Further, Beijing’s apparent anxieties about American chatbots aren’t unfounded. In a recent experiment, the Argument’s Kelsey Piper (a former Vox writer) presented various LLMs with 15 questions based on the World Values Survey, in a variety of different languages. She discovered that, even when prompted in Chinese, ChatGPT tended to express broadly left-of-center, anti-authoritarian views — and gamely provided advice on how to protest the government.
AI labs should still make sure their models aren’t getting oneshotted by Xi Jinping Thought
This does not mean that the major AI labs should shrug off these findings. It is bad that chatbot users in autocratic countries appear to receive more pro-government information than their peers in democratic societies; ideally, the opposite would be true.
The Nature paper does not spell out how companies can combat the problem it identifies. Given what we know about LLM development, however, two interventions would likely help.
First, during the pre-training phase — in which models independently glean patterns from large bodies of text — the labs could screen the most propagandistic forms of state media from their training datasets.
Second, during the “post-training” phase — when labs reprogram their models to substitute judgement for pure pattern matching — the companies could find ways of discouraging models from parroting autocrats’ talking points, in the same way that they currently deter them from providing tips on anorexic dieting or bioweapon development.
Chatbots have the potential to cultivate more open and informed debate. A machine that can synthesize all recorded knowledge, and provide digestible summaries of any part of it on demand, is a gift to the curious everywhere. And there is evidence that LLMs may be reducing the influence of misinformation and conspiracy theories, however marginally.
But the vast and growing power of the world’s biggest chatbots also presents profound dangers. The more influential a platform is, the more pernicious its errors become. Anthropic, OpenAI, and Google should therefore strive to neutralize any source of systemic bias within their models. Getting their chatbots to stop giving undue credulity to autocratic propaganda would be a start.