Planes grounded, television channels and banks malfunctioning, hospitals disrupted… Since the morning of July 19, many large companies and administrations have been running at a slow pace. Windows computers and servers are stuck on a “blue screen” of critical failure as soon as they start up. The fault lies in a faulty update of the Endpoint Detection and Response (EDR) software from the American company CrowdStrike, called Falcon.
To counter malware, EDRs intervene at a very low level in the operating system
EDRs are the successors to the antiviruses of the 90s, which detected signatures in executable files. If the antivirus found a match between the contents of a file and the signatures in its database, then it might be infected. An EDR goes much further: it performs behavioral tracking, records all system activities and stores them in a log, and can intervene at a very low level, all the way down to the operating system kernel.
These capabilities became necessary with the advent of increasingly sophisticated malware, which made antivirus obsolete. They are also what made possible this global outage when Falcon Sensor (the EDR component that performs very low-level monitoring) received a misformatted update, which caused its driver to crash when the operating system started.
EDR users often allow EDR vendors to perform automatic updates. This prevents systems from becoming vulnerable if the user is reluctant to accept the updates, and helps ensure the security of very large fleets of devices (the endpoints affected include servers, work computers, smartphones, and connected objects). This automation is the reason why thousands of companies were affected simultaneously.
Weekend updates are bad practice
It should be noted in passing that CrowdStrike played badly from start to finish. Because in terms of IT security, performing updates at the end of the week – and even more so in the evening – is an unwise choice: mobilizing IT teams on a Friday evening or a weekend is tricky and means that a breakdown or a security incident can last longer. This is why Microsoft always applies its monthly security updates on Tuesday morning.
Who is CrowdStrike?
Yet this is not CrowdStrike’s first rodeo. Appreciated by American institutions, the Texas company has been providing cybersecurity services to businesses since 2011. Its EDR software, Falcon, was launched in 2013. It has a large team of researchers, but also investigators who look into cyberattacks. In 2014, it discovered North Korea’s involvement in the cyberattack once morest Sony Pictures, and helped the FBI in its investigation into Russia’s hacking of the American Democratic Party.
How might this update be sent to customers?
Luckily, only Windows machines were affected, but that might have been the case. Falcon is also available for Mac and Linux, but the update in question only affected Windows. This is not the first time that an incident of this type has occurred with CrowdStrike solutions. A problematic update was released a few weeks ago for Red Hat Enterprise Linux distributions, but without much impact because it was not applied automatically.
What remains to be determined today is how such a flawed update was able to be sent to all CrowdStrike customers in this manner. Was it properly tested before its full deployment? The company will have to provide an explanation for this. George Kurtz, CrowdStrike’s founder and CEO, who spoke on X (formerly Twitter), has so far offered no explanation (or even an apology).
Can CrowdStrike recover from this incident?
It also remains to be seen whether CrowdStrike will be able to recover from this incident. On Wall Street, the company’s stock plunged more than 20% at the opening. Given the vendor’s past problems, will its customers remain loyal? Will they ask for compensation?
The total loss of revenue from this incident will in any case be in the billions of dollars, because while the machines can now be restarted through manual intervention, restoring the hundreds of thousands of systems affected might take time. This outage might therefore be one of the longest-lasting IT incidents in history.
Selected for you
