ProtonMail’s Bunker Datacenter | Image: ProtonMail
The ransom note arrived in the middle of the night, and it didn’t seem like a big deal. “We let users know we had been hit, called up the ISP, and then went back to sleep,” said ProtonMail CEO and CERN-alum Andy Yen. “Usually these guys hit you a few times, then move on, so you just ignore them.”
What came next was impossible to ignore.
Less than twelve hours later, on the morning of Wednesday, November 4th 2015, “things were out of control,” said Yen. The Switzerland-based secure email company ProtonMail was hit by one of Europe’s largest distributed denial of services (DDoS) attacks. ProtonMail servers were slammed with a 50 Gigabit per second wall of junk data that threatened to sink the company.
ProtonMail is not merely a technology startup. Born from work done at CERN, the research laboratory responsible for operating the particle-smashing Large Hadron Collider, ProtonMail’s mission is to develop and deploy end-to-end encrypted email and communication tools.
Similar to Lavabit, the email provider Edward Snowden used to communicate with journalists, ProtonMail is used to send secure communication around the world. “Lives of reporters, dissidents, and whistleblowers depend on the reliability and security of encrypted email services like ProtonMail,” said Frederic Gargula, co-founder of IP Max, the Geneva-based internet service provider (ISP) that helped defend ProtonMail during the attack.
ProtonMail uses end-to-end encryption to ensure that even the company itself does not have access to user messages. All data encryption and decryption happens on the client side, and data is secured using a passphrase which the company does not possess. In addition to providing strong security, this helps assure the company cannot share user data with third parties, or scan user data to serve targeted advertisements. Said Yen about the security of the platform, “if ProtonMail is ever breached in the future, only encrypted data would be accessible to the attackers.” And without the passphrase, he said, user data is useless.
Though DDoS attacks are common and crude, they are also effective at disrupting critical networking infrastructure. “When companies like ProtonMail are attacked, the entire security community is threatened,” said Gargula.
ProtonMail was actually hit by two groups of attackers, but on Wednesday morning, said Yen, it was not clear who was attacking. The first group, allegedly a group known as Armada Collective, sent a 15-minute test attack that arrived shortly after demanding a ransom of nearly $6,000, to be forked over using the semi-anonymous cryptocurrency Bitcoin. This ransom request was initially ignored by the ProtonMail team.
“At 2pm there was a dramatic escalation,” said Yen. According to Yen and Gargula the attackers smacked 15 different ISP nodes simultaneously, then attacked all the ISPs going into the datacenter using a wide range of sophisticated tactics. “This was not your standard DDoS anymore,” Yen explained, “in fact most of the experts we spoke with had never seen anything like this.”
An email similar to the ransom note sent to ProtonMail | Image: ProtonMail
Gargula agreed, “when I got to the office on Wednesday I knew we were going to be attacked, but my first priority was to support our customers. I had no idea how intense the fight would be.”
“First we moved the BGP IP prefix,” said Gargula as he detailed the attack, “I tried to isolate legit human traffic from bot traffic and not to mix it up. We sacrificed one of their three BGP uplink layers as a ‘canary’ to test the sophistication of the attack. Then we changed the configuration for the IP uplink.”
The new attackers were incredibly advanced, Gargula explained, and became more sophisticated through the week. “Every time we made a change in tactics, they responded with a change,” he said. “It was like Chess: you move a piece, they move a piece. At this point, it became clear that we had a very serious situation on our hands.”
The second attack was devastating and had a wide-ranging impact. “The second attackers impacted many other companies,” said Yen. “They also lost access to mission critical infrastructure.”
Regional data map of ProtonMail DDoS attack | Image: IP Max
ProtonMail’s primary datacenter was knocked offline completely, and the regional ISPs were struggling to stay up. “The collateral damage by then was hundreds of companies, with some as far away as Moscow,” said Gargula.
At this point, impacted companies quickly put pressure on ProtonMail to end the attack and all impacted companies got together for high level talks regarding paying the ransom. “From the start, ProtonMail has always been opposed to paying [ransom],” Yen explained, “but after discussions with other impacted companies, and considering the sheer amount of collateral damage, we respected the decision to pay.”
ProtonMail made an understandable but crucial mistake by paying the ransom, said Tim Matthews, vice president of marketing at Imperva Incapsula. The company was under attack and responded in a way they thought would mitigate the damage. However, explained Matthews, “once identified as an organization that will pay others may catch wind and come your way.”
However, what happened with ProtonMail was far worse. After paying the ransom, the attacks continued unabated. Running out of options, ProtonMail contacted the Swiss government agency MELANI and opened a dialogue with other companies recently hit by extortion attacks. After comparing the notes, the bitter truth sank in.
“We realized we were dealing with a different, far more scary attacker. One that didn’t fit the pattern of any previous attack,” said Yen. “At this point we knew there were two attackers. And the one attacking us now didn’t care about financial gain or fame, but whose sole objective was to kill ProtonMail at any cost.”
At that point Yen’s team knew they were at war.
The attack continued through the week, and the logistical challenges were as high as the stakes. If his team failed, ProtonMail would lack the resources to continue operation and cease to exist. Yet to succeed Yen’s team had to mitigate the attack, fund the defense, and find a permanent solution. “It was daunting,” he said.
Yen’s team was exhausted but determined to fend off the barrage. They approached the problem as scientists, he said, and approached the problem methodologically and systematically. “One team broke off quickly to do research into the attack against us, another one went to find help from our users, and a third began discussions with our partners and friends in Switzerland.”
European data map of Protonmail DDoS attack Image: IP Max
They broke the puzzle into manageable tasks. “If the ISP or datacenter could be hit, it was clear that protecting it was not enough,” said Yen. They had to protect everything from the server rack, “all the way through to the main internet gateways, then somehow do the engineering, finance the tasks, and find the DDoS protection provider who could protect us,” he said, “all in the span of a couple days.”
ProtonMail was crowdfunded, is free, and had sparse resources for defense. ProtonMail’s small team had no dedicated network engineers, not enough budget to fend off the attack, and the datacenter was buried in a mountain bunker 114 kilometers from the main internet point of presence (PoP) in Zurich.
“To solve the financing issue we took to social media and set up our crowdfunding campaign,” said Yen, “and immediately users jumped in to support us. We were deeply touched by this, usually when a service goes down users are ask for their money back, but instead they gave us their money without knowing if we would ever be able to recover from this.”
Radware chipped in to help provide DDoS protection, and through an 18 hour marathon on Saturday IP Max, Frederic Gargula’s small company, worked tirelessly to connect the datacenter to the main uplink.
“We even had other network experts joining into the rescue,” said Yen, “guys that had built both the datacenter in Zurich and our datacenter were intimately familiar with how to rewire both facilities.”
Regional data map of Protonmail DDoS attack | Image: IP Max
Helping ProtonMail was risky as it could potentially expose other companies to the data-wrath of the new attackers. A Google engineer in Zurich drove to the datacenter to perform a cross-connect. Level 3 Communications stepped in to provide an emergency IP transit. “It wasn’t just these guys that helped,” said Yen, “all of Switzerland pulled together to help us. Many datacenters and network specialists offered us assistance, fully knowing the risks of helping.”
“We helped because [ProtonMail] are like us,” said Frederic Gargula, “they are a part of our networking and security community. They care about encryption, and they protect user privacy.”
Finally, in the early hours of Sunday, November 8th, 2015, after a lengthy struggle, the joint ProtonMail, IP-Max and Radware team managed to turn the tide. Attacks continued a week after the assault, but were mitigated.
“A week-long DDoS is still pretty long, and we can’t rule out the possibility that they are regrouping,” said Yen, reflecting. “So we are staying vigilant. It is possible they are looking to change tactics now since we’re very well protected against conventional attacks.”
The second group of attackers still concerns Gargula and Yen. “DDoS is fast becoming, if not already, a commodity tool,” said Robert Morton, Akamai’s director of public relations. “It doesn’t take a lot of money, or necessarily skill to launch what could be a devastating attack. That means that any online business could be a target.”
Tod Beardsley, Security Research Manager for Rapid7 agrees. “DDoS tools are pretty common,” he said, adding that the tools to perform a massive attack like the one experienced by ProtonMail can be used by state actors like North Korea, or organized groups within permissive states like Russia. Sadly, he said, the culprits could be anyone.
Chillingly, “the second attacker has never made any demands or publicly taken credit,” said Yen, describing the second, more massive attack wave. “Their sole intention was to keep ProtonMail offline.”