Our DC was intact, but the building and access was cut-off. We lost the backup diesel power generators in the flooding. Of course, grid power was cut-off.
Our DC operating team managed to shutdown all the servers and racks cleanly before UPS power was completely drained. The 4 engineers and 2 security guards then swam out of the compound in chest high waters. (I am not kidding).
When the rains subsided and the flood waters receded after a couple of days, we had to plan the restart. The facility still had to be certified by health and safety, but we needed to get the datacenter back up.
A secondary operations site that would remote-connect to the DC was brought up in 1 week since we estimated the rains to potentially continue for a few more days and cause interruptions. But the critical item for the plan to work was getting a new backup power setup. We rolled in a truck-mounted diesel generator and positioned it in the highest point in the campus (also close to our building tower that had the DC) and ran power cables to it (we had to source this and it was a challenge to do it with the time crunch and the rains).
We moved staff to other cities by bus (airport was shutdown) as part of our recovery plan, but we still needed connectivity to our DC for some of the critical processes.
Long story short, it worked.
I'll never forget the experience and the scars from this war story.
