CrowdStrike is revamping its internal review process for software updates following a major outage caused by faulty code, which crashed millions of Windows devices worldwide. This change was announced in a blog post on Wednesday.
Microsoft’s estimate underscores the significance of this outage that 8.5 million Windows devices were affected after CrowdStrike's faulty update to its endpoint detection tools. Experts have called this incident the most significant IT outage in history, highlighting companies' heavy reliance on single vendors. The impact was widespread, with airlines cancelling flights through Monday, hospitals rescheduling non-urgent procedures, local 9-1-1 systems going down, and even Starbucks' mobile ordering service being disrupted.
CrowdStrike’s blog post provided a detailed explanation of the incident. The faulty update was meant to resolve a potential malware threat. It was pushed out early Friday morning using the company's "Rapid Response" protocols. This protocol typically involves quickly deploying a tested software update to all customers to mitigate cyber threats. However, the update contained problematic code that passed validation and triggered an "out-of-bounds memory read," causing the infamous Blue Screen of Death on affected devices. The resulting Windows operating system crash could not be handled gracefully, leading to the widespread outage.
In response, CrowdStrike is now adding another layer of testing to its processes and will stagger the release of such updates across customers to prevent similar incidents. Despite the severity of the outage, a significant majority of customers are now back online. CrowdStrike has been working closely with Microsoft to resolve the issue, and there have been no discussions about reducing CrowdStrike's access to Windows systems.
However, preventing another catastrophic outage is a challenging balance. Adding too many layers of internal testing could slow the response to potential threats, while insufficient testing risks another incident. Security experts have praised CrowdStrike for the detailed explanation in their blog posts. However, some customers feel more could be done to make amends.
In an attempt to address the situation, CrowdStrike sent $10 UberEats gift cards to partners, but some of these cards did not work because Uber flagged them as fraud because of high usage rates. A CrowdStrike spokesperson clarified that these gift cards were sent to teammates and partners assisting customers during the crisis, not directly to customers or clients.
Commentaires