I wanted to sleep at night. That is the unglamorous reason I did this.
When I stepped into this environment, we had disaster recovery. On paper. What that actually meant was we had tested failing over exactly one thing, the financial application. We ran that test every year, and every year it passed. But that one application was the whole of it. The domain controllers, the certificate authority, the backups, the networking, both data centers, the entire VMware environment, none of it had ever been failed over and proven. And there was no plan for it. We had the infrastructure, two data centers, a second site sitting ready, the tools to make it work in theory. What we did not have was a procedure. Nobody had ever written down how you actually recover this environment, and nobody had ever run it. The capability was there. The plan was not.
That sat badly with me, and it got worse the longer I looked at it. We were telling everyone, and ourselves, that we could recover. Meanwhile the only thing we had ever actually recovered was one application. The question I couldn't put down was the basic one. If something real happened, could we take the hit and come back? I did not want to be answering that for the first time in the middle of an actual incident, with the whole organization waiting on me.
I inherited this environment, and on paper there was every reason to leave it where it was. The box was checked. The annual test passed, the way it always did. Nobody was asking me to go poking at the rest. It would have been easy to take the green checkmark at face value. But a checkmark on one application is not the same as knowing the whole company can come back, and inheriting something doesn't let you assume it works. You kick the tires yourself. I really did not want to be two years in, sitting in a live outage, watching the recovery come apart and realizing no one had ever proven it would actually work.
So I decided we would do it for real. A full production failover of the whole environment, around seventy machines, everything from the high-impact production tier down to the low-priority back-office workloads, not just the one app. It would be the first time the company had ever attempted that, and we set it for this past winter.
It was an easy sell, which surprised me a little. When I laid it out, it was obvious to everyone. The logic makes itself: we don't actually know if this works, and the cheapest time to find out is on a schedule we choose, not during an emergency. Everybody agreed.
That didn't make it small. You are failing over real production. Get it wrong and you cause the exact outage you're trying to protect against. So we planned it hard, sequenced it carefully, timed it around payroll cycles so we had clean boundaries, and kept a way back if it went sideways. It still ran longer than we planned, because we kept tripping over things in the environment that nobody knew were broken.
That is the part I keep coming back to. The unknown unknowns.
Some of them surfaced before we ever cut over. Early on, working through the network architecture, we found that entire VLANs were missing on the VMware side at the Oregon site. They were spanned between the two data centers, one in Las Vegas and one in Oregon, at the network layer, but they had never been created on the ESXi and vCenter infrastructure on the far side. If a real disaster had been the first time we learned that, the machines would have had nowhere to land. One of our network engineers built them out before we went anywhere near a live failover.
Then there was the failover tool itself. Site Recovery Manager is VMware's product whose entire job is to orchestrate exactly this, and we had bought it years earlier and never deployed it. This was the first time anyone had actually stood it up. The week before we were set to cut over, it stopped replicating machines. Support eventually traced it to a time skew on four of the ten hosts at the second site, just enough clock drift to break the security tokens replication leans on. It was also quietly filling the datastores with delta files because it couldn't keep pace. We corrected the clocks on those four hosts, but the real lesson had already landed. The expensive tool we were counting on was the least reliable thing in the room. So we ran the actual failover on Veeam, which we already trusted, and it moved faster than we'd planned for. SRM did not survive the next budget.
And then the environment underneath it. The VMware estate was an inherited pile of small wrongs, none of them fatal on its own. A version drifting toward end of life. A mix of distributed and standard switches where it should have been one. Orphaned configuration scattered across hosts and vCenter. Host settings that did not even match between the two data centers, so the place we were failing over to was not a true mirror of the place we were leaving. Machines named and dropped into datastores with no logic anyone could reconstruct. So we straightened it out. Matched the two sides, fixed the naming, and gave the whole thing an order it never had.
None of it had surfaced in that single-application test, because that test never touched any of it. These were quiet failures sitting in the environment, waiting for the one day we would need everything to work at once. We only found them because we went looking.
One problem didn't surface until we were live. Partway through the failover, off-hours, application access dropped. The service accounts those apps use to sign in, the managed ones whose passwords the directory hands out, weren't working at the second site. We caught it, traced it, and had access restored before it turned into anything. AppDev and the help desk were validating right alongside us the whole way, which is the only reason something like that gets found and closed the same night instead of three days later.
There was nothing fast about any of it. Seeding every production VM over to the second site took two weeks on its own. We worked out an order of operations for bringing machines back up in a sane sequence instead of all at once, and even the failback ran long, because the changes that pile up while you're living on the far side don't just sync themselves home. And we documented every bit of it. Real runbooks, written steps, so recovery doesn't live in one person's memory. If we ever have to do this for real, or if someone does it after I'm gone, there is a tested procedure to follow instead of a guess.
By the end, the recovery plan was something we had actually run, with the broken parts found and fixed and the steps written down. I trust it now, because I have watched it work.
A recovery plan you have never run is just a hope with good formatting. It looks identical to the real thing right up until the moment you need it, and that is the worst possible time to learn the difference.
Inherited or not, it becomes yours the moment you're the one responsible for it. Nobody was going to make me test it. I would just rather find the broken parts on a Tuesday I picked than at 2 a.m. during the real thing.
And these days, I sleep fine.