Groot said:
I mean at that size, you'd expect some better backups/DR capabilities to be in place?
The key failing was not following industry standard procedure.
Their production system, backups and (supposedly) offsite backups were all on the same system and location & with the same access, thereby negating the entire benefit of offsite backups - that being physical distance between the two + lack of operator access.
I would imagine it was a cost-saving mechanism, but WHC really should have contracted through a 3rd-party cloud provider and then upload daily backups of customer data that were NOT accessible to even inside WHC people. That's what offsite is, storage that is far, far away from the production environment in case of fire, flooding, intrusion, theft, employee malfeasance, etc.
These offline data companies (like BackBlaze) simply store data, and there is no external access to delete, and if someone were to contact them and say "I'm from WHC, please delete all our data", even with the correct credentials, they'd tell them to FU. Their job is to protect data, not just let any Tom, Dick or Harry run rampant deleting offsite backups.
WHC staff or operator access should not have control over deleting or changing WHC's offsite backups, only production and local backups. There needs to be a big wall between production/local and offsite, or it doesn't work.
By having production, backup and (cough) offsite backup data effectively on the same systems & user access, as well as at the same basic location/area, there were effectively no backups in case of disaster, as everyone involved at WHC quickly found out.
