The easy call that night was to do nothing and point at someone else. It would have held up fine.
July 19, 2024. It hit late. The first sign was PRTG, our monitoring, lighting up as machines started dropping one after another. That alert rolled to our MSP, and the MSP started calling. So it began the way the bad ones usually do, a phone going off well after hours and a voice on the line telling you systems are going down and nobody knows why yet. Then the news caught up, and the news was staggering. Delta was cancelling flights by the thousand and would end up stranding passengers for the better part of a week. Sky News couldn't get on the air in the UK. Hospitals like Mass General Brigham and Memorial Sloan Kettering were postponing procedures and going back to paper charts. The 911 system in New Hampshire went down across the entire state. The same blue screen everywhere, all at once. A botched CrowdStrike update had gone out overnight and taken down around 8.5 million Windows machines worldwide in the space of a few hours.
And the scale of it is what got me. I have been doing this a long time, and I had never watched a single bad file take down that much of the planet overnight. It was the kind of thing you assume can't actually happen, right up until you're standing in it.
It wasn't a hack, which somehow made it stranger. CrowdStrike runs down at the kernel level of Windows, deep enough to watch everything, and to keep up with new threats it ships small content updates constantly. That night one of those updates, a sensor configuration file, went out with a flaw in it. The check that was supposed to catch a bad file let it through, the sensor reached for a piece of data that wasn't there, and the machine fell straight into a blue screen the instant it loaded. Every Windows box running their software and powered on in that window caught it. CrowdStrike found the mistake and pulled the file back a little over an hour later, but by then it didn't matter. If your machine had already taken the update, yanking it from the cloud did nothing. The bad file was already on your disk, and your disk would not boot.
Which handed us a very comfortable excuse. This was CrowdStrike's bug. CrowdStrike's fix to ship. Every headline said so, and the official posture from more or less everyone was to wait for the vendor's remediation. If I had walked into leadership and said the whole world is down, this is out of our hands, we're waiting on the vendor like everybody else, not one person would have blamed us. It was true. It was defensible. And it asked nothing of us but patience.
I could not get comfortable with it. We're a public agency, the kind a whole region quietly leans on, and "we're waiting on a vendor" is a sentence that ages badly when staff can't work and the days start stacking up. The fix itself existed. People had the cause figured out within hours, and the steps were almost insultingly simple: boot the machine into Safe Mode, open the CrowdStrike driver folder, delete the one bad file that started with C-00000291, reboot. Done. The whole problem was getting there.
We didn't run BitLocker, and that turned out to be the one mercy of the night. For a lot of the world, BitLocker is exactly where this turned into a multi-day ordeal. Getting an encrypted machine into Safe Mode meant punching in a 48-digit recovery key, one per device, and plenty of shops had those keys stored on servers that were now blue-screening themselves. We were spared that. What we weren't spared was the boot loop. Our machines were crashing the second Windows loaded the bad file, over and over, fast enough that you couldn't get a foot in the door to reach the recovery menu and ask for Safe Mode at all. The insultingly simple fix was sitting right there, behind a door that kept slamming shut.
So instead of sitting on our hands waiting for a cleaner answer to show up, we went looking for one. The CrowdStrike forums had turned into a live war room overnight. Technicians from everywhere were in there trading attempts in real time. This worked on a Dell, this didn't, here's what finally got me past BitLocker, try this on the loop. A few thousand strangers reverse-engineering a recovery out in the open while the vendor caught up.
Buried in all that was the thing that cracked it for us. Someone had found that going into the BIOS and flipping Secure Boot off was enough to knock the machine out of its loop on the next start and drop it into the startup options the blue screens had been racing past. From there you could send it into Safe Mode. On encrypted machines the same trick doubled as a way around the recovery-key prompt, which is why the forums were lit up about it, but we didn't need that part. We just needed off the loop. We tried it on one machine. It worked. Flip Secure Boot off, catch the boot into the recovery options, get into Safe Mode, delete the bad C-00000291 file, turn Secure Boot back on, and the machine came up clean.
Then we did it again. And again. There is nothing elegant about recovering an environment one BIOS menu at a time, but it worked every single time, and it was a real procedure instead of a hope. We worked through the machines, got people back to their desks, and brought the agency back up while a good part of the world was still parked behind that same recovery-key prompt, waiting on a fix to be delivered to them.
What I keep from that day is really about defaults. When something this big breaks, there's enormous gravity pulling you toward the passive version of the job. It's the vendor's fault, it's the vendor's fix, log the ticket and wait. And sometimes waiting genuinely is the right answer. This wasn't one of those times. The cause was understood, the fix was knowable, and the only thing between us and a working office was effort and a little nerve in someone else's BIOS settings. Pointing at CrowdStrike would have been completely accurate. It just wouldn't have turned a single computer back on.
Nobody handed out credit for it. There's no headline for the agency that quietly got itself back online on a day the news was busy with the ones that couldn't. The win was invisible, the way the best operations work usually is. We didn't wait to be rescued. Most days, that's the whole job.