Months before the Russian invasion, a team of Americans fanned out across Ukraine looking for a very specific kind of threat.
Some were soldiers, with the US Army’s Cyber Command. Others were civilian contractors and some employees of American companies that help defend critical infrastructure from the kind of cyber attacks that Russian agencies had inflicted upon Ukraine for years.
The US had been helping Ukraine bolster its cyber defences for years, ever since an infamous 2015 attack on its power grid left part of Kyiv without electricity for hours.
But this surge of US personnel in October and November was different: it was in preparation of impending war. People familiar with the operation described an urgency in the hunt for hidden malware, the kind which Russia could have planted, then left dormant in preparation to launch a devastating cyber attack alongside a more conventional ground invasion.
Experts warn that Russia may yet unleash a devastating online attack on Ukrainian infrastructure of the sort that has long been expected by western officials. But years of work, paired with the past two months of targeted bolstering, may explain why Ukrainian networks have held up so far.
Officials in Ukraine and the US are careful to describe the work of the “cybermission teams” as defensive, compared with the billions of dollars of lethal weapons that have poured into Ukraine to fight and kill Russian soldiers.
Russian attacks have been blunted because “the Ukrainian government has taken appropriate measures to counteract and protect our networks”, said Victor Zhora, a senior Ukrainian government official.
In the Ukrainian Railways, the team of American soldiers and civilians found and cleaned up one particularly pernicious type of malware, which cyber security experts dub “wiperware” — disabling entire computer networks simply by deleting crucial files on command.
In just the first 10 days of the Russian invasion, nearly 1mn Ukrainian civilians escaped to safety on the rail network. If the malware had remained undiscovered and was triggered, “it could have been catastrophic”, said a Ukrainian official familiar with the issue.
A similar malware went undetected within the border police, and last week, as hundreds of thousands of Ukrainian women and children tried to leave the country, computers at the crossing to Romania were disabled, adding to the chaos, according to people familiar with the matter.
With a much smaller budget — about $60mn — these teams also had to lay the ground with private groups that provide the backbone for most of the infrastructure that Russian hackers, either government-affiliated or not, were expected to attack.
On the last weekend in February, the Ukrainian national police, alongside other Ukrainian government arms, were facing a massive onslaught of “distributed denial-of-service attacks” (DDoS), which are relatively unsophisticated attacks that take down networks by flooding them with demands for small amounts of data from a large number of computers.
Within hours, the Americans had contacted Fortinet, a Californian cyber security group that sells a “virtual machine” designed to counter just such an attack.
Funding was approved within hours and the US Department of Commerce provided clearance within 15 minutes. Within eight hours of the request, a team of engineers had installed Fortinet’s software on to Ukrainian police servers to fend off the onslaught, said a person familiar with the rapid-fire operation.
The fact that these onslaughts are often targeting commercially available software — mostly from western manufacturers — has forced major US and European companies to dedicate resources to defending Ukrainian networks.
Microsoft, for instance, has for months run a Threat Intelligence Center that has thrust its resources in between Russian malware and Ukrainian systems.
On February 24, a few hours before Russian tanks started rolling into Ukraine, Microsoft engineers detected and reverse-engineered a newly activated piece of malware, Microsoft’s president Brad Smith has said in a blog post.
Within three hours, the company issued a software update to protect against the malware, warned the Ukrainian government about the threat and alerted Ukraine about “attacks on a range of targets”, including the military. On the US government’s advice, Microsoft immediately extended the warning to neighbouring Nato countries, said a person familiar with the late-night decision.
“We are a company and not a government or a country,” Smith wrote, but added that Microsoft and other software makers needed to remain vigilant against what happened in 2017, when a malware attributed to Russia spread beyond the borders of the Ukrainian cyber arena to the wider world, disabling computers at Merck, Maersk and elsewhere and causing $10bn of damage.
So far, experts who have watched the Russian cyber assaults have been confused at their lack of success, as well as the lower tempo, intensity and sophistication of what Russian-government hackers are known to be capable of.
Ukrainian defences have proved resilient, said one European official who was briefed this week by the Americans at a Nato meeting, and Russian offences have proved mediocre. He said the reason was that, so far, Russia has held back its elite corps in the cyber arena, much as it has on the battlefield, perhaps by underestimating the Ukrainians.
One example, he said, was the fact that instead of communicating solely through encrypted military-grade phones, Russian commanders are sometimes piggybacking on Ukrainian cell phone networks to communicate, at times simply by using their Russian cell phones.
“The Ukrainians love it — there is so much data in simply watching these phones, whether or not they are using encrypted apps,” he said.
The Ukrainians then block Russian phones from their local networks at key moments, further jamming their communications. “Then you suddenly see Russian soldiers grabbing cell phones off Ukrainians on the street, raiding repair shops for sims,” he said. “This is not sophisticated stuff. It’s quite puzzling.”