But it's just kind of interesting, you can have all the redundant systems and smart software and some dude could accidentally pull cables – oh humans!
Would love to see what other mitigations they came up with than the ones listed (apart from probably putting 20 BRIGHT RED labels next to the patch panels saying DO NOT DISCONNECT, EVER EVER EVER!).
Perhaps one mitigation could be a better way to literally identify who's there and call them up within seconds and ask what they just did?
So they failed to label their cables? I'm sorry but this is "datacenter 101" stuff. How are none of the cables plugged into your patch panels labeled? Every colo has a label gun you can borrow! Also remote hands will gladly send you a pic of a rack or cabinet to verify what they're looking at.
> we knew that the failback from disaster recovery would be very complex
The disaster recovery failover to a second data center (and failback) should not force a choice to failover or not. They should be able to immediately failover and the system should self-heal once the original data center was back online.
Submarine Cables are like this too. It all comes down to a quarter inch thick bunch of fibers (each being thinner than a human hair)