On 19 July 2024, the American cybersecurity company CrowdStrike distributed a faulty update to its Falcon Sensor security software that caused widespread problems with Microsoft Windows computers running the software. As a result, roughly 8.5 million systems crashed and were unable to properly restart in what has been called the largest outage in the history of information technology and "historic in scale".
The outage disrupted daily life, businesses, and governments around the world. Many industries were affected—airlines, airports, banks, hotels, hospitals, manufacturing, stock markets, broadcasting, gas stations, retail stores, and governmental services, such as emergency services and websites. The worldwide financial damage has been estimated to be tens of billions of dollars.
Within hours, the error was discovered and a fix was released, but because many affected computers had to be fixed manually, outages continued to linger on many services.
Background
CrowdStrike produces a suite of security software products for businesses, designed to protect computers from cyberattacks. Falcon, CrowdStrike's endpoint detection and response agent, works at the operating system kernel level on individual computers to detect and prevent threats. Patches are routinely distributed by CrowdStrike to its clients to enable their computers to address new threats.
CrowdStrike's own post-incident investigation identified several errors that led to the release of a faulty update to the "Crowdstrike Sensor Detection Engine":
Outage
On 19 July 2024 at 04:09 UTC, CrowdStrike distributed a faulty configuration update for its Falcon sensor software running on Windows PCs and servers. A modification to a configuration file which was responsible for screening named pipes, Channel File 291, caused an out-of-bounds memory read in the Windows sensor client that resulted in an invalid page fault. The update caused machines to either enter into a bootloop or boot into recovery mode.
Almost immediately, Windows virtual machines on the Microsoft Azure cloud platform began rebooting and crashing, and at 06:48 UTC, Google Compute Engine also reported the problem. The problem affected systems running Windows 10 and Windows 11 running the CrowdStrike Falcon software. Most personal Windows PCs were unaffected, since CrowdStrike's software was primarily used by organisations and not SOHO technology infrastructure. The CrowdStrike software did not provide a way for subscribers to delay the installation of its content files. Computers running macOS and Linux were unaffected, as the problematic content file was only for Windows, but similar problems had affected Linux distributions of CrowdStrike software in April 2024.
CrowdStrike reverted the content update at 05:27 UTC, and devices that booted after the revert were not affected.
At 07:15 UTC, Google stated that the CrowdStrike update was at fault. Within hours, CrowdStrike CEO George Kurtz confirmed that CrowdStrike's faulty kernel configuration file update had caused the problem. At 09:45 UTC, Kurtz confirmed that the fix was deployed and that the problem was not the result of a cyberattack.
The impact to companies in the Central United States was exacerbated by an unrelated outage with Microsoft Azure the previous day. On 18 July, the Azure platform had an outage that blocked some companies' access to their storage and to Microsoft 365 applications in Azure's Central United States region.
Remedy
Affected machines could be restored by rebooting while connected to the network, ideally while connected to Ethernet, thus providing the opportunity to download the reverted channel file, with multiple reboots reportedly required.
If crashes persisted, remediation required booting into safe mode or the Windows Recovery Environment and deleting any .sys file beginning with C-00000291- and with timestamp 04:09 UTC in the %windir%\System32\drivers\CrowdStrike\ directory. As this process needed to be done locally on each individual machine, it was "expected to take days" for affected businesses to restore all systems. Technical staff needed to reboot the affected computers individually with manual intervention on each system.
On devices with Windows' BitLocker disk encryption enabled, which corporations often use to increase security, the problem was exacerbated because the 48-digit numeric Bitlocker recovery keys (unique to each system) required manual input, with additional challenges supplying the recovery keys to end users working remotely. Additionally, several organisations utilising local servers for Bitlocker recovery key storage could not access keys that were stored on servers that themselves had crashed.
Microsoft also recommended restoring a backup from before 18 July to fix the issue.
Impact
Outages were experienced worldwide, reflecting the wide use of Microsoft Windows and CrowdStrike software by global corporations in numerous business sectors. At the time of the incident, CrowdStrike said it had more than 24,000 customers, including nearly 60% of Fortune 500 companies and more than half of the Fortune 1000. On 20 July, Microsoft estimated that 8.5 million devices were affected by the update, which it said was less than one percent of all Windows devices.
Widespread outages were immediately reported across multiple countries, with major global disturbances experienced by the general public. At 04:09 UTC on 19 July, the time when the faulty update was issued, it was the middle of the business day in Oceania and Asia, the early morning hours in Europe, and midnight in much of the Americas.
Some countries were less affected. China, which has striven toward self-sufficiency in IT, saw no impact on its daily services including airlines and banks, although some foreign branch companies and luxury hotels in the country were affected. Russia and Iran—both restricted by international sanctions from using the services of American high-tech companies—reported no disruptions.
Cyber risk quantification company Kovrr calculated that the total cost to the UK economy will likely fall between £1.7 and £2.3 billion ($2.18 and $2.96 billion).
Specialist cloud outage insurance firm Parametrix estimated that the top 500 US companies by revenue, excluding Microsoft, had faced near $5.4bn (£4.1bn) in financial losses because of the outage, but only between $540m (£418m) to $1.08bn (£840m) of those losses would be insured.
CrowdStrike liability
CrowdStrike's own terms and conditions for their Falcon software limit liability to "fees paid", effectively a refund. Larger customers may have negotiated different terms.
In the EU, it is possible that CrowdStrike will be held liable under a GDPR regulation related to the impact of security incidents on user data. The regulation is best known in relation to data leaks but also applies to data destruction. It is unclear whether temporary loss of access to data is enough to trigger liability, or whether GDPR applies to all incidents related to security or only unauthorised access.
Further, the incident could be classed as a "personal data breach" which would be a data breach of the GDPR under Article 4, "Definitions", paragraph 12. On 19 July 2024, a data-protection expert reported a breach of Article 32, "Security of processing".
Air transport
Globally, 5,078 air flights, 4.6% of those scheduled that day, were cancelled. An unrelated Microsoft Azure outage, affecting services such as Microsoft 365, compounded airlines' problems.
Oceania
Australian airlines Qantas, Virgin Australia, and Jetstar were affected. A Sydney Airport spokesperson said that the outage had affected some operations and that "there may be some delays throughout the evening". Melbourne Airport saw check-in procedures disrupted; officials advised passengers to consult with their airlines. The Adelaide, Brisbane, Canberra, Darwin, Hobart, Launceston, and Perth airports were also affected. In New Zealand, Christchurch Airport also had problems.
Asia
Hong Kong International Airport experienced delays during check-in, primarily for passengers of the local budget carrier Hong Kong Express, whose staff members used handwritten signs to direct passengers to check-in counters. The Hong Kong Airport Authority activated an emergency response after airline websites and automatic check-in malfunctioned. The booking systems of local airlines Cathay Pacific, Hong Kong Express, and Hong Kong Airlines were unavailable. HKExpress cancelled some flights on 20 July. Jeju Air and Spring Japan experienced problems. Jetstar Japan cancelled many (mostly domestic) flights. Some of the self-check-in kiosks in Singapore Changi Airport were affected, delaying and forcing airlines to switch to manual check-in, and Singapore Airlines and Scoot reported service difficulties on 19 July. Cebu Pacific and Philippines AirAsia flights were delayed. Long queues formed at Ninoy Aquino International Airport. In Taiwan, airline system disruptions were reported at Taoyuan International Airport. In Indonesia, disruptions were reported for the check-in systems of AirAsia and Citilink. In Thailand, Thai AirAsia's reservation and check-in systems were affected.
In India, the outage affected IndiGo, Akasa Air, SpiceJet and Vistara. Handwritten boarding passes were issued throughout the outage. The Ministry of Civil Aviation ordered airlines and airports to be compassionate and provide food and seating to waiting customers as needed. At 18:14 IST (12:44 UTC), over 200 Indian flights had been cancelled; IndiGo alone had cancelled 192. Airlines that relied on Microsoft Azure for their services were affected. Air India and SpiceJet stated that none of its flights were cancelled due to the outage, attributing it to their robust cyber system. However, minute delays were reported.
Europe
Prague Airport in Czechia, Budapest Airport in Hungary, Bratislava Airport in Slovakia, and Schiphol Airport in the Netherlands experienced problems. Planes were barred from landing at Zurich Airport. Employees of Charleroi Airport, near Brussels, manually checked passengers in, but other software alleviated problems by 10:00 (UTC+2) and there were minimal delays. ENAIRE's Aena, the Spanish national airport traffic control manager, mentioned an IT outage on their website and social media. All Spanish airports reported disruptions. Charles de Gaulle Airport and Orly Airport experienced check-in problems and suspended flights. Poznań–Ławica Airport and Warsaw Chopin Airport experienced check-in disruptions. An emergency system was activated, and check-in processes were slower. Berlin Brandenburg Airport announced that since around 07:00 (UTC+2), operational processes had been affected by "IT problems at an external provider", and that they planned to stop flights until 08:00 UTC. While passenger handling continued with some restrictions, there were delays and airlines cancelled some flights. Several airlines (Eurowings, Ryanair, Vueling, and Turkish Airlines) in Hamburg Airport had to issue tickets by hand. Croatian and Swedish air traffic control were also disrupted.
Swiss International Air Lines had 30% of flights grounded. Lufthansa in Germany experienced problems with the "profile and booking retrieval" features of their website. Ryanair's booking and check-in services were unavailable and the airline was "forced to cancel a small number of flights", advising passengers to arrive at airports at least three hours before departure. Wizz Air said the outage put its online services offline. Dutch airline KLM suspended most operations, announcing that flight handling was impossible with the issue, and Transavia Airlines experienced problems. Finnair reported that they were having trouble sending emails and SMS messages to customers. In Greece, citizens and tourists saw delays at major airports, notably at Athens International Airport and at Heraklion International Airport. This disruption, occurring at the peak of the tourist season, resulted in chaotic scenes as passengers were forced to wait for hours for their flights. Contributing factors included severe staff shortages and new schedules. In Heraklion, eight flights were problematic. The airport's chief, George Pliakas, indicated that flights were being manually arranged to manage the disruption, but the influx of arriving flights strained the system.
Several UK airports had problems, including Edinburgh Airport, whose departure boards froze, and Gatwick Airport, where automatic barcode scanning stopped working and tickets had to be checked manually. Amadeus, which manages UK baggage at Heathrow, said they were affected by the IT outage. Disruption to flights was anticipated in the Isle of Man, particularly to and from the UK, but ultimately minimal.
Middle East and North Africa
Tunisia experienced temporary airport disruptions. Turkish Airlines cancelled some of its flights due to the outage.