How a New Law Is Making It Difficult for Russia’s Aggregators to Tell What's New(s)

Just past noon on March 26, a headline appeared on the homepage of Yandex, Russia’s largest search engine: “Anti-Corruption Protests Held in Major Russian Cities.”

Soon after, it disappeared. For hours, while police dragged hundreds of protesters off Moscow’s streets and pushed them into detention vans, Yandex’s homepage listed its top stories as: Russia’s Eurovision contestant was denied entry to Ukraine, the Yemen president was sentenced to death, and a Duma deputy’s obscure aide committed suicide.

In the media industry, a fast news cycle is a given. But to many, Yandex’s turnover seemed excessively fast. Increasingly wary of Kremlin meddling, some have accused one of Russia’s most successful tech giants of deliberately burying news of the anti-government protests to mollify the authorities.

Yandex has denied manual meddling and points to its algorithms. Four experts — including two former employees of the company — told The Moscow Times they find the explanation credible, but that it is only half the story.

The incident highlights the effect of a new law that came into effect this year, which some say has turned algorithms into the effective accomplices of state censorship. “You don’t need manual intervention to block news that the government doesn’t want you to see,” says Anton Nossik, a tech entrepreneur often called the godfather of the Russian internet.

Internet in aggregate

What aggregators do matters. Russia’s second and third largest news aggregators — Mail.ru and Rambler — receive 4 and 1.6 million daily visitors respectively, data from the TNS market research company shows. But with roughly 6 million daily users, Yandex’s news service, Yandex.News, is the undisputed leader.

Consequently, the platform largely shapes the country’s online news consumption. Even those who mainly use Yandex as a search engine (roughly 12 million people daily), are served the five biggest news stories on its homepage, right above the search bar.

“Anyone with internet will visit Yandex’s homepage before doing anything else,” says Artyom Kozlyuk, head of the Roskomsvoboda internet watchdog. That makes the top five list “a sensitive issue for the authorities,” he says.

However, if a story does not make the top five, the implication is that it is not big enough. This made the lack of exposure given to the March 26 protests especially painful.

According to the Vedomosti paper, the story of Russia’s largest protests in years never made the top five at all while they were in full swing. Elsewhere on the portal, the news was unusually difficult to find, it claimed.

Alexei Navalny, who led the protests, lashed out in anger on Twitter. “I’m ashamed on Yandex’s behalf today,” he tweeted, calling any explanation short of a confession of manual meddling on state orders a lie.

But in a statement admitting it had “lagged behind the picture of the day,” Yandex described the distortion as a technical problem. It said its algorithms automatically select top stories based on a topic’s popularity and the authority of the sources covering it. Since January, however, those sources have become its main headache.

New state control

After first allowing the internet to exist as a relatively free space, in recent years the authorities have steadily increased their control, clamping down on bloggers and placing new obligations on internet companies. It was always a matter of time before aggregators would be targeted, too. The “news aggregators law,” which went into effect in January this year, initially proposed to hold aggregators with an audience larger than 1 million daily users liable for all content on their websites — an impossible burden, according to the aggregators themselves.

As a compromise, aggregators could bypass the requirement by limiting their source material to media officially registered with Russia’s media watchdog, Roskomnadzor. They had little choice but to comply, says Kozlyuk from Roskomsvoboda.

“They are fighting for their survival,” he says. “The alternative is constant checks and millions of rubles in fines.”

In practice, Yandex.News has undergone a purge. It dropped unofficial outlets such as blogs and foreign media. It has also scrapped sources such as Meduza — a site based in Latvia which is among the most authoritative out of a handful remaining independent Russian-language news sources.

In the past, the aggregator considered providing Russians with a balanced selection of news as one of its main tasks, says Tatyana Isayeva, who headed Yandex.News for four years, until resigning over the law in 2016.

It would link to primary sources on blogs or foreign news websites to faciliate fact-checking. Now such practices are no longer possible. And by definition, what Yandex.News shows is content that has either been approved, or directly funded, by the state.

“Certain aspects of life that are covered by outlets like Meduza simply do not exist anymore for Yandex’s algorithms,” says Isayeva.

The implications became visible on March 26. State media largely ignored the protests or referred to them only as “unsanctioned meetings,” likely reducing the events’ relevance in the eyes of Yandex’s bots, the company suggested.

Quantity, not just quality, provides another challenge for Yandex’s algorithms, which naturally assign weight to frequently cited sources and topics.

Since government portals and state-affiliated websites were allowed to register as media, aggregators have been flooded with “white noise.” According to Yandex.News statistics, roughly 551 of its sources are “official” sites. This especially impacts local news feeds.

During an election campaign for the Moscow Duma in 2014, Muscovites found their feeds swamped with news from unknown local outlets detailing the successes of the city’s mayor. And, whether intentionally or not, news of the March 26 protests was effectively pushed out of Muscovites’ local news feed by reports on an upcoming spring festival and warnings of icy weather.

“The task of an aggregator is to use a huge amount of data and good algorithms to answer two questions,” says Isayeva. “What important topics are the media writing about? And where can I read about it best? [Yandex] is struggling with both.”

Getting it right

Most experts contacted by The Moscow Times said that technology could overcome the restrictions placed on aggregators — in theory. Yandex itself said it would investigate what went wrong with its March 26 coverage and look for technical solutions.

In the meantime, Russians can bypass attempts to regulate content by using features which allow users to limit their news feed to sources they trust, like Yandex’s Zen. And, for now, Google News has avoided Roskomnadzor’s restrictions because it is less popular.

But as long as the pressure on Russia’s media sphere remains in place, the prospects for Russian aggregators are grim, says Isayeva.

Aggregators are a reflection of the general mediasphere, she says. “At the moment, their main problem is that they cannot add anything to the picture painted on [state] television,” she adds. “Technological glitches can be fixed, but, in essence, the service has become useless.”