Embarrassingly for the Kremlin, Russian AI Isn’t Good Enough for Its Own Disinformation

Last Friday, OpenAI, the company behind ChatGPT, revealed the first confirmed use of generative AI as a tool of Russian disinformation. The campaigns the company disclosed were not very effective, garnering merely hundreds of views and engagements across all platforms. But the most humiliating thing for Russia was not the campaigns’ lackluster returns, but the fact they were reliant on an American LLM.

The previously discovered Doppelganger targeted a wide multilingual audience with realistic clones of news websites generated using OpenAI’s models. Meanwhile, the newly discovered Bad Grammar targeted Russian speakers in the Baltics and Moldova, generating Telegram comments in English and Russian and debugging code for a bot on the platform. Both campaigns bore signs that ChatGPT was used for everything from content generation to translation to coding. 

Russia has invested heavily in its own AI chatbots which until recently outperformed competitors in the Russian language. But when Russian operatives needed high-quality Russian content for semi-official government purposes, they looked to American software to provide it. The episode reveals that despite increasingly dire talk from Russian leaders about the threats posed by foreign AI, Russia is falling ever further behind, even in its mother tongue.

Russian leadership was among the first to recognize the revolutionary potential of AI. “Artificial intelligence is the future not only of Russia but of all of mankind,” Putin declared in 2017. His administration was one of the first in the world to release a comprehensive AI strategy, and the Russian government has invested tens of millions of dollars in building up its domestic industry.

But by last year, excitement had turned to fear as American AI trounced Russian competitors like Yandex’s Alice and Sber’s GigaChat. Kremlin leadership sought to paint American AI as a threat to Russian values. Putin explained at an AI summit that Western models “reflect that part of Western ethics, those norms of behavior, public policy, to which we object.” His Prime Minister declared Western models had “a different understanding of good and evil.”

To the Kremlin, promoting Russian AI is about more than solving intractable engineering problems, optimizing code, or creating and spreading disinformation. It is about preserving and protecting Russian culture, values, and ways of life from an onslaught of Western-biased systems. But the administration’s attempts to bolster domestic AI labs so they at least compete in their own language seems to have bore little fruit.

As recently as a year ago, tech bloggers in Russia, while overall preferring GPT-3, spoke highly of GigaChat and Alice’s ability to carry on longer Russian conversations. Earlier this year, Sber released a new version of GigaChat which performed on par with OpenAI’s GPT-3.5 on a battery of Russian-language questions.

But GPT-3.5 is over a year old by this point. Its latest successor, GPT-4o, performs far better. According to Chatbot Arena, a forum where users choose the best answer to a given prompt from two anonymized chatbots, OpenAI’s latest model provides a better answer in Russian than its predecessor 81% of the time. GPT-4o is not even the best model in Russian, losing out (though just barely) to both Google’s Gemini Advanced and Anthropic’s Claude 3 Opus.

The scoreboard, however, does provide some hope for non-Western alternatives for Russian-language bots. However, it would relegate the former superpower to an increasingly familiar position: the junior partner of China. Chinese startup 01 AI’s model ranks sixth overall in Russian and Zhipu AI ranks 16th. Both are ahead of GPT-3.5 and, by inference, GigaChat. 

Putin promised to dedicate his chairmanship of BRICS to developing a “counterhegemonic” AI governance structure. But nearly halfway through the year, he has little to show. Most of the bloc’s output has come from China, which recently released an AI evaluation index by country. The index showed that China has 50 times as many AI professionals, 10 times as many AI patents, and 239 times the funding for AI as Russia. In each of these categories, Russia was behind each of the original BRICS nations with the exception of South Africa. Meanwhile, the United States was ahead of China, often by a factor of 2 or more.

Generative AI benefits from network effects: the latest AI is used to create the next generation. That means whoever has better AI this generation can use it to improve faster. American AI should only continue to expand its lead over its Russian counterparts. Even without this effect, China’s index shows that the United States invests over 600 times as much in the technology as Russia.

As the Bad Grammar disinformation network shows, this gap has widened since Sber and Yandex released their GPT competitors last year. The result is that Russians are almost certain to use American or — in the best case for Russia — Chinese models. Putin’s fears that such models could erase Russian culture and history may not be enough to overcome the fact that non-Russian offerings are far more sophisticated. When looking for assistance with basic tasks, Russians, like everyone else, are likely to go with convenience and competence over nationalist bravado. That bodes poorly for Putin’s dreams of AI supremacy.