I have to laugh at all of you who are all in on the "a good data center is foolproof! They must have cheaped out!" Two real world examples, both Microsoft Azure: One time they were doing cleanup and had a script to delete old SQL VM instance backups. Unfortunately the script was written wrong on instead of deleting "old backups" it deleted the live customer VMs. Every SQL instance running on the whole East Coast node went poof. Oopsie. That was a three hour outage for my company. Another real case, they somehow had a whole rack of management servers that was totally outside all the fault tolerance mechanisms. No backups, no secondary power supply, not even scripted into the VM management systems. And then they had a power failure, and all the actual customer machines were fine, they failed over fine, but this rack of network routers, VM managers, etc went down hard. And when it got back on power and rebooted, there was nothing to tell the master VM manager that these servers were supposed to be core infrastructure, and it just saw a bunch of high power servers and started spinning up high price tier customer VMs on it, overwriting all the critical VMs that weren't backed up. That one didn't hit my company, but it was a 24+ outage for a lot of Azure customers.
