Blue screen of debt

A firm with $3.1bn in annual revenue has just stopped about 8.5m computers.^{2MicrosoftHelping our customers through the CrowdStrike outage20 Jul 2024,4CrowdStrike (SEC filing)CrowdStrike Holdings, Inc. annual report on Form 10-K for the year ended 31 January 20247 Mar 2024} That is Microsoft’s estimate, published on 20 July, for the Windows machines that crashed after CrowdStrike, a cybersecurity company, sent out a faulty update the day before.^{2MicrosoftHelping our customers through the CrowdStrike outage20 Jul 2024} Microsoft is at pains to note that the figure is less than 1% of all Windows devices.^{2MicrosoftHelping our customers through the CrowdStrike outage20 Jul 2024} It is a comfort of the sort offered by the man who points out that most of the plane was fine.

The facts are simple. CrowdStrike said the outage was caused by “a defect found in a Falcon content update for Windows hosts”, and that “this was not a cyberattack”.^{1CrowdStrikeStatement on Falcon Content Update for Windows Hosts19 Jul 2024} Macs and Linux machines were untouched.^{1CrowdStrikeStatement on Falcon Content Update for Windows Hosts19 Jul 2024} America’s cyber agency, CISA, was aware of “the widespread outage affecting Microsoft Windows hosts due to an issue with a recent CrowdStrike update”.^{3CISAWidespread IT Outage Due to CrowdStrike Update19 Jul 2024} No hacker did this. A security product, installed to keep the lights on, switched many of them off.

• Revenue
• Annual recurring revenue

A small firm, a large footprint. CrowdStrike, fiscal years ending 31 January, $bn

	Revenue	Annual recurring revenue
FY2023	2.2	2.6
FY2024	3.1	3.4

Source: CrowdStrike annual report (10-K)⁴

We think the lesson is not that CrowdStrike is uniquely careless, which will be settled by those who investigate it, but that it holds a kind of power the market does not price. Security software of this type sits deep inside the machine, with the access to inspect everything because its job is to stop anything. The same access means a bad update can take the machine down with it. A firm with 29,000 subscription customers is trusted with the keys to a good deal of the world’s offices, hospitals and departure lounges.^{4CrowdStrike (SEC filing)CrowdStrike Holdings, Inc. annual report on Form 10-K for the year ended 31 January 20247 Mar 2024} Its own annual report, filed in March, concedes that its software “may contain defects or errors that are not detected until after deployment”.^{4CrowdStrike (SEC filing)CrowdStrike Holdings, Inc. annual report on Form 10-K for the year ended 31 January 20247 Mar 2024} Investors were told. Passengers were not.

The best defence of the status quo deserves a fair hearing. Security is a race, and the best protection comes from a handful of firms that see the most threats. Concentration, on this view, is a feature: if one vendor sees an attack on a bank in Singapore, every customer in Chicago is protected within minutes. Firms that spread their business across many vendors would each see less of the threat. Microsoft’s own point, that the outage hit a small share of devices, is fair as far as it goes.^{2MicrosoftHelping our customers through the CrowdStrike outage20 Jul 2024}

It does not go far enough. A diversity of vendors protects against a bug in one product only if the products fail separately. Here the failure came through an update channel that reaches every customer at once. The speed that makes a security vendor useful is the same speed that makes its mistakes global. In ordinary infrastructure the answer is well known: power grids, airlines and banks all stage their changes, test on a small group first and keep the ability to roll back. Software that can ground aircraft should meet the same standard, even if it began life as a subscription.

There is a second asymmetry. A customer can test the software it installs, but an update that arrives by itself is another matter. The buyer carries the cost of the vendor’s mistake, and the vendor carries the cost of its reputation. Those are not the same size of bill.

So we propose three modest things. Vendors with kernel-level reach should be required to roll out updates in stages, and to say publicly how. Regulators of banks, airlines and hospitals should ask their regulated firms to prove they can recover from a vendor’s mistake as well as an attacker’s. And the liability that vendors have disclaimed should be revisited, at least for failures that a small test group would have caught. None of this requires deciding that CrowdStrike is a villain. The firm was quick to say it was not an attack. What it will say about the next update is the more interesting question.

Firms were told to guard against the unknown intruder. Few thought to ask who was guarding the guard.