Ofir Dor, TheMarker
(Originally published in Hebrew on TheMarker.com, Sept 13, 2024)
It was 10:00 AM on Saturday, December 12, 2020. Tim Brown, CISO of American software company SolarWinds, received a dramatic call. On the other end of the line was his boss, then-CEO Kevin Thompson, who shared astounding news: "I just spoke with Mandiant, and they say we're sending our customers manipulated versions of our software.”
"I was in complete shock," recalls Brown, 60, in an interview with TheMarker. "There really wasn't any doubt left, as Mandiant had good evidence that we were sending out versions with malicious code in them. They showed us the code from within our own software, and we could clearly see that there was a segment that was simply not our code. Mandiant made it clear to us that the information about the incident had already leaked through other channels, and would becoone public the following day - so we had to prepare a statement immediately to be released the next day."
Brown: "The magnitude of the event was understood quite quickly. In the first statement we published the next day, we already mentioned that 18,000 of our customers had downloaded manipulated versions. That's a large number, so we knew it was a critical incident that would affect many. It was a difficult period. In the first month, I didn't sleep. I lost 11 kilograms in three weeks."
The SolarWinds incident, dubbed the SUNBURST attack, turned out to be one of the largest and most sophisticated cyberattacks in history, and perhaps the largest and most sophisticated of them all. The attack was discovered during the COVID-19 pandemic, when SolarWinds employees were working from home, but the company immediately understood that they could not communicate using the company's email system, which was being monitored by the attackers. Instead, they spoke on the phone or communicated through external emails only.
The first task for Brown and his team was drafting a report on the attack for the SEC, the U.S. Securities and Exchange Commission - and they only had a day and a half to do so. Submitting a file on an event of this magnitude which is at such an early stage proved to be such a complex task that SolarWinds CEO later joked that every comma in the filing cost the company $20,000 in legal fees.
After the announcement, all hell broke loose. SolarWinds’ switchboard lit up like a Christmas tree, receiving 19,000 calls on Monday. Concerned customers demanded to know what they needed to do, and governments around the world asked if organizations in their territory had been affected. Revealing the attack caused the company's valuation to plummet from $7.4 billion to $4.5 billion in just a few days. CISA, the U.S. Cybersecurity Agency, Brown says, was mainly concerned if any of the coronavirus vaccine development labs might have been breached.
I felt like a burglar had broken into my home, rummaged through, and moved my stuff around. It felt really bad," says Brown. "The second feeling was responsibility of what our customers are going through. We’ve met with a lot of angry customers. You have to understand that people are going through things similar to what you're feeling - and not get mad about it. They, too, are experiencing pain because they need to find out whether they were affected, and they're not with their families. You have to develop very thick skin, because people won't say nice things about you for a long time.
When SolarWinds’ people started investigating what exactly happened to them, they found that the attack on the company had begun almost two years prior, in early 2019, when attackers who stole an employee's credentials logged-on to the company network from an external computer. The attackers went in and out of SolarWinds’ network over months, gradually gaining more and more access permissions to new sectors, downloading some 7 million emails from 70 mailboxes of company executives and employees, copying source code and parts of the system, and making their way to the ultimate target.
3,500 Lines of Code
SolarWinds is a medium-sized company from Austin, Texas, that develops network management software. The company was not particularly interesting as a target, but its customers, which included a long list of U.S. government agencies, tech giants, and think tanks dealing with U.S. national security issues, certainly were. Therefore, the SolarWinds attackers penetrated the company's holy of holies - the development process of its flagship software, Orion.
During the software build process, the attackers replaced one of the files and inserted in its place a file containing 3,500 additional lines of code. The new file was bundled with the Orion software - and sent to customers. It contained SolarWinds' digital signature, so it looked like an integral part of its software. Through this file, the attackers were able to penetrate the networks of some of the most secure entities in the U.S., without being subject to a security check upon entry. The attackers repeated the process during the build stage of three Orion versions.
This was a new grade of attack, and it was clear from the beginning that only nation state backed hackers could have accomplished such a mission. The Americans attributed the attack to SVR, Russia's Foreign Intelligence Service, the equivalent to the U.S. CIA and the Israeli Mossad.
Of approximately 18,000 customers who received the malicious file, SVR only penetrated about 100, but those included entities holding scads of sensitive information, such as the U.S. Departments of Justice, Homeland Security and Energy, and the U.S. Nuclear Energy Council. The full extent of the data stolen from these entities has not been made public and is not fully known to this day, but the hackers could potentially have accessed sensitive data such as nuclear missiles locations, confidential court files, and planned U.S. government sanctions. Microsoft reported that the hackers accessed parts of its source code; and Mandiant, which discovered the attack, reported they stole some of its cybersecurity tools.
"The attackers were very calculated," says Brown, "they didn't invent new technologies, but acted in a very thoughtful way. For example, after they entered a customer's system, they waited 14 days before initiating the attack. It’s not technically difficult to do, but it's a move that indicates a calculated approach. They were stealthy, they went in and out, did some trial runs first - and then waited a while before launching the attack in February 2020. By June they had already removed the attack. They defined their mission and aimed towards it.”
When such a state actor attacks you with all their resources, is there really any chance of defending against it?
“You can always improve, make it harder for them, and better recover from an attack. You can do a lot of things, but if someone like that targets you - it will certainly be very difficult to defend against."
Do you think they specifically targeted SolarWinds?
"Apparently."
The Missed Warnings
In the days following the attack, SolarWinds' stock plummeted, wiping out about 40% of its value. Overcoming the attack took months, but after SolarWinds finished the defense stage, they received another blow in the form of an indictment filed by the SEC in October 2023 against the company and Brown personally, accusing them of investor fraud. Reading the SEC indictment for the first time, it's easy to rush to blame Brown and SolarWinds of irresponsibility and negligence, at the very least.
The indictment details how in the months before the scandal erupted, two of the company's customers - first the U.S. Department of Justice and later Israeli entrepreneur Nir Zuk’s cybersecurity firm Palo Alto Networks - contacted SolarWinds and said their Orion software instance was inexplicably communicating with external sites. At the time, both incidents were classified as low importance at SolarWinds, and therefore not escalated by Brown to the CEO and CTO of the company, even though internal correspondence at SolarWinds referred to the issue as disturbing. When Palo Alto Networks asked SolarWinds if it had encountered a similar incident before, an employee at the company lied that it didn’t, failing to mention the inquiry from the Justice Department on the matter.
The indictment also presented a series of internal communications within SolarWinds, in which Brown and company engineers complained about security issues. In 2018, a SolarWinds employee highlighted security gaps in the company’s VPN remote access system, stating that anyone who gained access through it could "do almost anything they wish." The issue was not addressed, eventually becoming the vulnerability through which the Russian attackers infiltrated SolarWinds. The SEC indictment also accused SolarWinds of incomplete reporting to investors both before and after the attack.
However, last July the US District Court for the Southern District of New York dismissed most of the charges against SolarWinds and Brown. The court ruled that of all the charges, the SEC would only be able to proceed with the claim that SolarWinds published inaccurate information regarding its security status on its website, text on which Brown signed off.
The judge ruled that SolarWinds internally investigated the Palo Alto Networks and DoJ claims, and even if they were mistakenly classified as low importance, it did not constitute a systemic flaw as the SEC had claimed. Regarding SEC’s claim that Brown should have escalated the VPN security breach issues to a higher level, the court wrote that this “claim has traction only with the benefit of post-SUNBURST hindsight.”
90% of Charges Dismissed
"Since this is an ongoing legal process, I can only say that we are pleased that 90% of the charges were dismissed, but I can't go into any more details," says Brown, who continues to serve as the company's CISO, contrary to what one might expect after an incident of this kind, a fact that Brown attributes to his ability to adapt his skills to the new situation. "Many times, the replacement of security managers after a cyberattack results from the company suddenly requires someone with a different set of skills," he says.
Did you manage to change the perception that you were breached because your security wasn't good enough?
"I think so, and the proof is that our customer retention rate is 97%-98% today. Was our security good enough to deal with the Russian SVR? No, it wasn't. That's a fact. The SVR attacked us successfully. Was our security reasonable? We think so, and that's what we've stated many times. Now we're working hard to make our security exemplary."
You didn't cooperate with the media during the event. Was that a mistake?
"One of the comments journalists offered me after the fact was that I should have had media partners I could talk to and rely on. But if I had wasted time talking to the media, I wouldn't be helping our customers. There were good articles written about us and there were some that offered inaccurate details. It wasn't the fault of the reporters, they were talking to people who had worked at the company six or sometimes ten years earlier, and their view didn't reflect the current reality. In crises like this, there are always people who claim they told us in advance that this was going to happen. In hindsight, things always seem very clear."
Could attackers still be lurking somewhere in your customers' networks?
"The code the attackers left was very specific. Nowadays, the moment any security system in the world detects a file containing such code, it starts beeping immediately, so restarting the attack again would be almost impossible. But I'll bet you that right now, other organizations are in the same place we were. It would be naive to think the Russians chose to only pursue us, and that we're the only company they did this to."
Attacking Through a 4ᵗʰ Party
It's hard to underestimate the impact of the SolarWinds incident on the global cybersecurity industry. The attack was a key factor that led U.S. President Joe Biden to issue a special executive order in May 2021 requiring government agencies to improve their cybersecurity and the protection of government computer networks. The SolarWinds attack also gave rise to a long list of startups in the cybersecurity field, including many Israeli startups, that were created specifically to address the flaws that arose from it. "In the new world, it's not at all certain that you as an organization will be attacked directly, but they can attack you through a 3ʳᵈ party, or even a 4ᵗʰ party, which is a partner of one of your partners, and which you don't even know exists," says Brown.
Brown, who has gained a reputation as a man who experienced the biggest cyber attack and held to his job to tell the story, says that many Israeli entrepreneurs approach him to get his opinion on their cybersecurity ideas. "I'll tell them straight to their face that their idea will absolutely not work, and send them back to the drawing board. I've managed to change the minds of some of the entrepreneurs, especially those at an early stage. It's important to give them value, because our industry's goal is to defeat the enemy, and a lot of the innovation comes from Israel," he says.
However, ideas from Israel that he does believe in occasionally reach Brown, like that of the Israeli startup Cytactic, founded by Dr. Nimrod Kozlovski, former partner at Herzog Fox & Neeman law firm and a cybersecurity investor who has raised $16 million for Cytactic since its foundation in 2022, to develop a technological platform for managing cyber crises like the one experienced by SolarWinds. The platform walks organizations through building preparedness to cyber incidents, and in the event of a breach, serves as a crisis management virtual command and control center.
Brown was introduced to Cytactic when one of its investors asked him for his opinion on its solution before investing. Brown recommended it, and also became an external advisor to Cytactic - and just last week it was announced that Brown is joining the company's Advisory Board. "I've met a lot of security managers in my life who, unlike Tim, panicked when they were attacked. They tried to hide things and became defensive," Kozlovski compliments. "In Tim's case, I think the incident actually turned him into a leader in the security management community, because he acted with openness and transparency - and he's always happy to talk and exchange ideas.
The suggestions Brown made contributed to changes in Cytactic's product, for example, towards more knowledge sharing between different organizations using the software. "Everyone talks about information sharing in cybersecurity, but no one actually does it," says Kozlovski. "Companies are not eager to share information with other companies or with authorities, because they fear exposing their intellectual property or their weaknesses. They can't talk about the cyber incidents they've experienced, so everything remains secret. Tim said that our platform could be ideal for knowledge sharing between the different players. Companies can divulge their non-IP cyber crisis preparedness plans on the platform and contribute to the community. This made us understand that we need to give more weight in the platform to the companies’ ability to share their methods.
Today, almost four years after that Saturday when SolarWinds was attacked, Brown says he's trying to look at the positive things that grew out of that event. "I think I learned a lot from what I went through, and the incident gave me the opportunity to work with the best cyber security managers in the world," he says. "From the industry's perspective, the possibility of a state-sponsored cyberattack was something everyone was discussing theoretically - but few shared that they had actually experienced such an attack. We came forward and said, ‘yes, we were attacked by sophisticated actors, such actors do exist and are carrying out operations.’ As security managers, this has heightened our awareness of the risk.
