# Major incident at WHC.ca regarding hosting



## theinvestor__ (Aug 28, 2021)

This has been ongoing for most of the day. 

August 28, 2021: a major incident is impacting the availability of web hosting and reseller hosting accounts on several WHC systems in our BHS1 Montreal datacenter. We currently have all hands on deck working on the problem but the situation is serious.

See link below :

https://whc.ca/blog/live-major-incident-in-progress


----------



## MapleDots__ (Aug 28, 2021)

This is very serious

Any idea how everything got wiped?


----------



## MapleDots__ (Aug 28, 2021)

Ohh man, as I keep reading I feel for the team at WHC, such a tremendous loss.

They have not announced how the information was wiped from the servers so at this point we can only speculate.


----------



## theinvestor__ (Aug 28, 2021)

It really is inexcusable. It doesn’t matter what happened to these servers but to not have a backup is just bad news.


----------



## MapleDots__ (Aug 29, 2021)

Never count on webhosts, I know for DN CA I use a schedule and downloaded a complete backup daily. Most that could be lost is 1 day.

Anyone running a site can easily do that using cPanel

In fact while they were sorting things out I would already by running on a different server. Propogation time is pretty quick now.


----------



## theinvestor__ (Aug 29, 2021)

Update August 29, 2021 - 10:00AM

Thank you once again for your patience. We continued to work overnight on restoring accounts.

Here are the updates for each of our restored servers so far:
- Rev2 server has been fully recovered and should be functioning normally
- Decarie server has been fully recovered and should be functioning normally
- Rev server was recovered, however, some issues remain

Here are the updates for servers currently in the restoration process:
- Rachel's restoration process should begin within the next hour
- Peel restoration is approaching 30% completion.
- Beaubien restoration is approaching 50% completion.
- Atwater restoration is approaching 20% completion.

Services on the above servers should be fully functional within the next 12-24 hours.

Please note: We will be posting another update at 11AM for the more severely impacted servers.


----------



## Spex (Aug 29, 2021)

Total disaster..my email and a few sites are completely offline. The updates are making it seem pretty bad. I'm on the 'bernard' server and their not expecting any recovery at all



> Accounts on servers with low likelihood of data recovery
> The following 5 systems have had their external backup servers also partially destroyed, and we have been as of yet unsuccessful at recovering their data:
> - Clark
> - Drummond
> ...



How does a server get destroyed...fire? flood?


----------



## MapleDots__ (Aug 29, 2021)

Spex said:
			
		

> How does a server get destroyed...fire? flood?



They have not answered any of that, they need to be transparent if they want this to not seriously impact their reputation.

I was going to move DN.ca hosting there a few months ago but decided not too because I had prepaid hosting for 3 years.


----------



## MapleDots__ (Aug 29, 2021)

I wonder if Sibername and TBR will be affected by all of this


----------



## theinvestor__ (Aug 29, 2021)

Update August 29, 2021 - 11:00AM

Thank you everyone for your patience and support during what has been a very trying 24 hours. Our team has been working tirelessly to restore service to affected clients and have had some success but the news isn’t all good, so here is where we stand.

Servers that have been fully or partially recovered

We have successfully recovered the following servers:
- Decarie
- Rev2
- Rev (some data loss has been reported here)

Accounts on servers with high likelihood of data recovery

We have 4 systems whose external backup servers have not been impacted, and these are actively being used to stream account restorations right now. We expect to be able to fully restore all backed up accounts by Monday evening and hopefully sooner. Here is where they are in the restore process:
- Beaubien (50%)
- Peel (30%)
- Atwater (20%)
- Rachel (1%, restore has just been started)

A full restore process for a system of this size generally takes 24 hours. As soon as an account is restored, its website and email should become available so for most users on these machines you should have your services restored before the end of the day Sunday.

Accounts on servers with low likelihood of data recovery

The following 5 systems have had their external backup servers also partially destroyed, and we have been as of yet unsuccessful at recovering their data:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

For clients on these machines, we are pursuing 3 recovery strategies in parallel:

Re-upload your content from your own backup
If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
Partial data recovery on production servers
We have had some success in recovering hosting accounts data files (excluding databases) for 2 of the affected servers (Acadie & Bernard for now) and may be able to perform a partial restoration of these accounts within the next 48-72 hours. This process is slow and tedious and unlikely to complete before Tuesday.
Manual data recovery 
We’ve contracted an external firm to assist us with data recovery efforts on our backup servers and they started their work on Saturday night. Manual data recovery is a tedious and slow process, but we expect to get a report on what can be salvaged before day’s end. This approach will likely take the longest to result in recovered data, with a timeline as long as 1-2 weeks.
We will be posting more information along with a post-mortem once the immediate operational issues have been addressed.

We have mobilized our full team to assist in this effort and we currently have dozens of system administrators, support technicians and data recovery engineers assisting with the recovery effort since yesterday. We understand this incident may seriously impact your business and are doing everything in our power to mitigate the damage.

More information will follow as we have it.


----------



## theinvestor__ (Aug 29, 2021)

Update August 29, 2021 - 5:00PM

As our team continues to work to restore accounts, please note that live updates are paused until tomorrow morning at 10AM so we can again focus on the restoration efforts. We’ll then be able to give an account of what has been accomplished overnight.

Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:
- Beaubien is at 55%
- Peel is at 35%
- Atwater is at 30%
- Rachel is at 10%

Accounts on servers with damaged backups

The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

Our initial attempt to repair the data on our backup servers has failed and at this point the likelihood of successfully restoring account data from these servers is very low.

As a result, we are working with data recovery experts to attempt to restore data from the source server, but this process is extremely long and tedious, potentially requiring file-by-file analysis for systems with millions of files and may require months to complete.

Here is how you can move forward if your account was on one of those servers:

If you have a local backup
If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
If you don’t have a local backup: introducing “Lifeboat” accounts
To facilitate continued business operations for our clients during this major outage and until the situation is resolved, we will be activating temporary hosting accounts for all clients hosted on Clark, Drummond, Acadie, Bernard and Bishop servers. You will be able to log in to this new account in order to recreate your emails and upload website content, without hindering ongoing recovery efforts. A simple nameserver change will be required and step-by-step instructions will be provided. These accounts are expected to be available before Monday morning.


----------



## theinvestor__ (Aug 30, 2021)

Update August 29, 2021 - 11:00PM

Accounts on servers with damaged backups

With our initial data recovery process unsuccessful, we’re now recommending that all affected clients that have experienced data loss consider using a new, temporary hosting account that’s already created in their Client Area called LifeBoat.

To use this account, click into it to create your emails and upload website data just as you would on your regular hosting account.

Once this is done, just link your domain to this account, update your domain’s DNS to:

ark1.whc.ca
ark2.whc.ca

Once the DNS changes propagate (can take several hours) your new account can be used to start accepting and sending new email and upload new website copy quickly. These accounts will remain free of charge until at least January 1, 2022 and are intended as a stopgap measure until a more permanent solution is found.

Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:

- Beaubien is at 62%
- Peel is at 45%
- Atwater is at 45%
- Rachel is at 31%


----------



## theinvestor__ (Aug 30, 2021)

Update August 30, 2021 - 9:00AM

The restoration process is advancing for the 4 servers where backups are available:

- Beaubien is at 88%
- Peel is at 65%
- Atwater is at 65%
- Rachel is at 64%

For clients on servers with damaged backups, we recommend you initiate your disaster recovery plan ASAP. We have created hosting accounts in your Client Area called "LifeBoats" to help you get up and running while we explore long-term data recovery options. The impacted servers are:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop


----------



## jaydub__ (Aug 30, 2021)

Wow…this is a certified mess


----------



## MapleDots__ (Aug 30, 2021)

The million dollar question is what happened and can the company survive this.

Even the fact that [notify]FM[/notify] has not commented indicates something pretty big.


----------



## jaydub__ (Aug 30, 2021)

I am thinking I should get my stuff out of Siber, although at this point they seem to be ok.
This is why I was hoping in the takeover, whc wouldn’t try (and glad they haven’t at this point anyway) and mess with Siber’s already pretty successfull TBR system.


----------



## FM__ (Aug 30, 2021)

MapleDots said:
			
		

> The million dollar question is what happened and can the company survive this.
> 
> Even the fact that [notify]FM[/notify] has not commented indicates something pretty big.



You've been posting all of the updates here - at this moment I can only refer you here:
https://whc.ca/blog/live-major-incident-in-progress/


----------



## MapleDots__ (Aug 30, 2021)

FM said:
			
		

> You've been posting all of the updates here - at this moment I can only refer you here:
> https://whc.ca/blog/live-major-incident-in-progress/



Well, it is the days of the internet

This is the data centre WHC uses

https://www.datacenterknowledge.com...-green-hydro-powered-facility-in-quebec-opens


Now lets investigate and see what we can find


----------



## theinvestor__ (Aug 30, 2021)

Major Incident: What happened?

Share this article    
 Emil Falcon
 August 30, 2021

WHC News & Events
It’s been a tough weekend here at WHC and by this, I include our clients. I want to start by thanking all the team for coming together and working through the problem constructively and with tremendous heart and energy.

Here’s the situation.

Based on our investigation to date, the morning of August 28 at approximately 6AM, an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers.

Within only hours our incident response team had identified the issue and disabled access to the source account, preventing any further damage. The environment was secured, the individual fully locked out, and our disaster recovery plan immediately kicked into action but damage was already done.

The tally of the incident, however, was and still is important: a few major systems, including some production servers and some backup servers were damaged, with a large number of web hosting and reseller hosting accounts affected, resulting in possible permanent data loss.

After nearly 2 days of tedious work and a combination of external datacenter backup restores and system-level storage rebuilding, our team was able to successfully recover (or is in the process of recovering) over 50% of those lost accounts. We can confirm that Cloud, Dedicated, Weebly and Managed WordPress accounts were largely unaffected.

Unfortunately, at the moment, I can tell you that several production servers and their respective backup servers are still in an unrecoverable state and the data recovery experts assisting us believe that the recovery potential is very low. As such, the focus for these accounts has shifted from data recovery to starting fresh on new, clean accounts. In parallel we will continue to attempt to recover any data we can.

For clients impacted by this incident and for which we are unable to recover backups:

If you have an adequate local backup: contact our support team and we will get you up and running on a new account ASAP
If you do not have a local backup: You will need to start over from a bare, empty account. To this end, we have activated new, free hosting accounts for each impacted domain, called LifeBoat. They are available in your Client Area now and are intended to serve as a quick, free and immediate way for you to start over. These LifeBoat accounts will remain free of charge until at least January 1, 2022.
On behalf of WHC, I would like to extend our sincere apologies to those affected by this unfortunate situation. With the cooperation of those particular clients affected by the incident, we believe that we can greatly minimise the consequences stemming from this involuntary incident.

We remain committed as ever to providing you with quality and affordable hosting solutions. We understand and regret the impact that this incident may have on your business and operations.

We are also grateful and moved by the outpouring of support we have received as we continue working to tackle this issue.

Sincerely,

Emil Falcon
CEO at Web Hosting Canada


----------



## MapleDots__ (Aug 30, 2021)

theinvestor said:
			
		

> Based on our investigation to date, the morning of August 28 at approximately 6AM, an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers.



Server reimaging - does that not mean trying to initiate a backup from a previous server image?

If that was the case the server should be in a restored state to an earlier date right?

Not sure if this provides any answers, was it malicious intent or was it an accident.

WHC needs to be fully transparent with this and then state how an event like this can be prevented in the future. Anything else will cause uncertainty and probably an exodus of clients.

In all plain English, what happened and how can it be prevented from ever happening again.


----------



## theinvestor__ (Aug 30, 2021)

MapleDots said:
			
		

> Server reimaging - does that not mean trying to initiate a backup from a previous server image?
> 
> If that was the case the server should be in a restored state to an earlier date right?
> 
> ...



Well, considering they said the individual was locked out. Tells me it was malicious. It leaves a lot of questions on the table. 

1. Did this individual hack the server? If so how did they do it? 

2. If the servers were being copied, who has this information now?

3. What personal information got exposed? 

Sorry, but this is not a great update nor did it provide any major insight. 

Don’t forget in May we got an email that their systems got compromised. So since May they didn’t do anything to secure their systems? That was a sign to bail at that time…


----------



## MapleDots__ (Aug 31, 2021)

It hit CBC News


https://cybernews.com/news/a-major-incident-wiped-data-on-web-hosting-canada-servers/

Says WHC serves 60,000 clients and it looks like ransomware


----------



## MapleDots__ (Aug 31, 2021)

https://www.itworldcanada.com/article/web-hosting-canada-reveals-cause-of-outage/457684


----------



## theinvestor__ (Aug 31, 2021)

MapleDots said:
			
		

> It hit CBC News
> 
> 
> https://cybernews.com/news/a-major-incident-wiped-data-on-web-hosting-canada-servers/
> ...



Your eyes must be doing tricks on you. Interesting comments though….


----------



## MapleDots__ (Aug 31, 2021)

HeHe Cybernews

I could have sworn I saw a CBC logo when I looked at the news on mobile


----------



## theinvestor__ (Aug 31, 2021)

It’s interesting that WHC is distinguishing and mentioning that Sibername accounts had recoverable backups. I don’t get it. Was WHC not doing backups?


----------



## MapleDots__ (Aug 31, 2021)

Lesson for everyone here, if you leave your backup in the webhosts hands you are in trouble.
It only takes a minute to generate a full account backup and then download it.

When was the last time you did one?


----------



## theinvestor__ (Aug 31, 2021)

You know, WHC is going to lose some business over this I am sure. But to be honest this is not something that occurs often and even the web hosts that mention that they do backups they cannot guarantee it for you. Their TOS covers their butt. So, it really leaves the onus on the customer to do their own backups. 

I had my own server and I backed up offsite daily. That’s just how a business should operate. If you’re running a website and email you have to pay to back up. It’s a cost that you have to accept. I hope WHC can recover smoothly from this.


----------



## MapleDots__ (Sep 2, 2021)

Received an update which specifically says this was NOT a ransomware attack




> Dear WHC Client,
> 
> As some of you may already know, WHC suffered a major incident Saturday morning, impacting a subset of clients with web hosting, email hosting and reseller hosting accounts. We want to write to you today to provide an update.
> 
> ...


----------



## MapleDots__ (Sep 2, 2021)

Copy of the letter...


----------



## theinvestor__ (Sep 2, 2021)

Sometimes if you catch something early enough it doesn’t get to ransomware. It doesn’t mean it wasn’t…and you can’t conclude it’s not without a proper investigation. That doesn’t happen in a week.


----------



## Groot (Oct 30, 2021)

I mean at that size, you'd expect some better backups/DR capabilities to be in place? Yeah, it's up to customers to back up their sites, yada yada, but still, this was pretty big.

Good on WHC though recovering as much as they did.


----------



## DomainRecap (Oct 30, 2021)

Groot said:
			
		

> I mean at that size, you'd expect some better backups/DR capabilities to be in place?



The key failing was not following industry standard procedure. 

Their production system, backups and (supposedly) offsite backups were all on the same system and location & with the same access, thereby negating the entire benefit of offsite backups - that being physical distance between the two + lack of operator access.

I would imagine it was a cost-saving mechanism, but WHC really should have contracted through a 3rd-party cloud provider and then upload daily backups of customer data that were NOT accessible to even inside WHC people. That's what offsite is, storage that is far, far away from the production environment in case of fire, flooding, intrusion, theft, employee malfeasance, etc. 

These offline data companies (like BackBlaze) simply store data, and there is no external access to delete, and if someone were to contact them and say "I'm from WHC, please delete all our data", even with the correct credentials, they'd tell them to FU. Their job is to protect data, not just let any Tom, Dick or Harry run rampant deleting offsite backups. 

WHC staff or operator access should not have control over deleting or changing WHC's offsite backups, only production and local backups. There needs to be a big wall between production/local and offsite, or it doesn't work. 

By having production, backup and (cough) offsite backup data effectively on the same systems & user access, as well as at the same basic location/area, there were effectively no backups in case of disaster, as everyone involved at WHC quickly found out.


----------



## MapleDots__ (Oct 30, 2021)

For DN.ca I use cPanel to do a full account backup EVERY single day and they download both to my google drive and my local computer. From there on they are stored for 90 days with the oldest one deleting when the new on comes in.

Since all our pictures are stored on ImagePost.org the entire backup with database is under 25mb. Our forum software is so slim and streamlined that a full restore takes under 5 minutes and we are up and running again.

So if we get hacked I can run a mirror site, fix the hack and bring the site back live very fast.

It is the responsibility of a good webmaster to do this and not depend on the webhost. It is easy to lay blame but really anyone that depends on a website and blindly leaves it up to the webhost is living in a dream world.

Had DN.ca been hosted on WHC we would have been up and running as soon as WHC moved us to a new server. Worst case there would have been a few hours propagation delay and the most we can ever lose is one days worth of posts if we have to restore from the previous day.

So as much as one would blame WHC, the responsibility also lies heavily on the lazy webmasters who have all the tools but refuse to take the time to use them.


----------



## DomainRecap (Oct 30, 2021)

MapleDots said:
			
		

> It is the responsibility of a good webmaster to do this and not depend on the webhost. It is easy to lay blame but really anyone that depends on a website and blindly leaves it up to the webhost is living in a dream world.



Sorry, but this is wrong, as they expressly stated in ads and other areas of the site "Offsite Backups" and although most people had local backups, some did not work (probably WHC screwed up the backup) and others were corrupted. Without offline backups, there was no way to try different ones.

Go read some of the comments on their FB page and you'll see plenty of people with local backups that simply would not work. I think WHC had problems well before this. 

Or what if the customer's backup HD blew up, system got stolen, or damaged by electricity, fire, or flooding? **** happens, that's why you pay for a host that states they use "offline backups".

Myself, I had newer backups downloaded, but the first few didn't work when uploaded and as this process took a week to work through, I was forced to go to an older "known good" backup just to get up and running. It was truly a nightmare and there is no way to 100% prepare for something like this when the host issues run so deep. 

What if all your recent backups are corrupted/unusable at the source, and then your host melts down totally with no offline backups? What emergency plan do you have?


----------



## MapleDots__ (Oct 30, 2021)

DomainRecap said:
			
		

> Sorry, but this is wrong, as they expressly stated in ads and other areas of the site "Offsite Backups" and although most people had local backups, some did not work (probably WHC screwed up the backup) and others were corrupted. Without offline backups, there was no way to try different ones.



Back in the day of Lunarpages I too believed my webhost when they said they did the backups. That was a mistake and I no longer trust any company to do the backups for me. In fact if a company did do them for sure I would still do mine just in case the company somehow got a ransomware attack or something.

For me having my own backup copies stored on a cloud and on a local computer makes sure I never get into a compromised position.


----------



## rlm__ (Oct 30, 2021)

MapleDots said:
			
		

> Back in the day of Lunarpages I too believed my webhost when they said they did the backups. That was a mistake and I no longer trust any company to do the backups for me. In fact if a company did do them for sure I would still do mine just in case the company somehow got a ransomware attack or something.
> 
> For me having my own backup copies stored on a cloud and on a local computer makes sure I never get into a compromised position.



To be honest, I wouldn't trust anyone with my backups either...


----------



## DomainRecap (Nov 1, 2021)

MapleDots said:
			
		

> For me having my own backup copies stored on a cloud and on a local computer makes sure I never get into a compromised position.



Once again:

*What if all your recent local backups on your HD & cloud are unusable due to being corrupted/unusable at the source,* and then your host melts down totally with no offline backups? What emergency plan do you have?

This is what happened at WHC for many people, who held locally stored backups that were corrupt. With nothing offline at WHC and their local servers and backups toasted, a lot of people who used your infallible strategy were totally screwed. 

Offsite backups by the host should not be the only solution for the end user (locally stored backups, multiple storage locations, etc.), but they are one more level of protection and flexibility should the crap really hit the fan. To not understand their importance in MIS is insane.

There were more problems at WHC than just the meltdown, and once again, as I posted in that same message, go to their FB "meltdown" post and see how many people are having trouble restoring local backups they have stored, often because the backup itself was corrupted at the source. 

I experienced this myself and was forced to use a "known good" (from testing) backup just to get back online.


----------



## DomainRecap (Nov 1, 2021)

rlm said:
			
		

> To be honest, I wouldn't trust anyone with my backups either...



Sure, if you run your own server you can have direct access to the backup generation tools and process, as well as testing each and every backup in a development environment, but I don't know of any commercial host that allows this kind of access.

With companies like WHC, you download the Jet backup file and trust them that it's not corrupted or otherwise unusable.


----------



## DomainRecap (Nov 1, 2021)

And guys, I apologize if I'm being blunt here, but I am a WHC customer and I was right there on the front lines experiencing it. 

And it was a nightmare, and as other customers noted, they had never seen a sheer hosting disaster like this, ever. 

I had recent backups and I scheduled a restore. Several days later (yes, it was a hosting holocaust and everything took forever) it would fail, and I would try the next-newest backup, and a day or two later it too would fail - rinse and repeat. By then it was impossible to get through to support (phones were off and email was clogged) but finally the theory was that the most recent backups were corrupted. Many others experienced the same issue. 

This was an absolute nightmare and every request took days to complete, and I did everything right. I had local backups, I had them stored locally, on a network server and also on the cloud. And I still got totally hosed and lost a pile of data, so for a couple of Monday Morning Quarterbacks on non-WHC hosting to tell me "what they would have done" and absolving WHC for a very significant lapse in procedure, is a tad irritating. 

This will be my final post on this matter.


----------



## MapleDots__ (Nov 1, 2021)

DomainRecap said:
			
		

> And guys, I apologize if I'm being blunt here, but I am a WHC customer and I was right there on the front lines experiencing it.



I feel your pain and am for sure not arguing with you.

All I am saying is I purposely seek out a host that has cPanel because they have super easy tools to allow you to backup and fully restore.

In a perfect world the Webhost does as expected but because I have run into a very similar issue with Lunarpages way on back I have since never relied on the webhost. For my system ideally we have a webhost backup, my local backup, and my cloud backup. Between the 3 I should never be caught in the rain.

So all I am saying is it's better to have 3 backup systems than 1 to assure redundancy.

I remember Lunarpages told me I had all the tools with cPanel, why did I not use them. They were absolutely right, I relied on them and they failed. After that I never again relied on a host. Sure I was glad they did backups but a part of me knew I could never be 100% reliant on that. That is where cPanel backup comes in. Simply backup the entire account, now you can upload the file to any cPanel host and be running as soon as propogation allows.


----------



## rlm__ (Nov 1, 2021)

DomainRecap said:
			
		

> f any commercial host that allows this kind of access.





			
				DomainRecap said:
			
		

> Sure, if you run your own server you can have direct access to the backup generation tools and process, as well as testing each and every backup in a development environment, but I don't know of any commercial host that allows this kind of access.
> 
> With companies like WHC, you download the Jet backup file and trust them that it's not corrupted or otherwise unusable.



Always get hosting with SSH access.  You can use tar and rsync to copy backups off to another server, and if transfers get interrupted, rsync can pick up where it left off rather than starting from the beginning again - ensuring backups are not corrupted during transfer.

I do have a couple of my own dedicated servers, and I back one up to the other - and vice versa, all done automatically using cron, tar, mysqldump and rsync.


----------



## DomainRecap (Nov 1, 2021)

As stated, I am certain these were not corrupted via transfer, but at the source.

There were problems at WHC before the official meltdown, which just exacerbated the damage.


----------



## rlm__ (Nov 1, 2021)

DomainRecap said:
			
		

> As stated, I am certain these were not corrupted via transfer, but at the source.
> 
> There were problems at WHC before the official meltdown, which just exacerbated the damage.



Just saying that network issues can cause transfers to crap out.  But yeah, if the source is corrupted, you're kinda screwed from the start.  You'd think there'd be error checking on backups though.

I'm just glad I've always handled my own backups as back in the early days I had multiple issues where I needed them due to hackers and overzealous colo facilities that shut down servers.  I had hackers inject malicious code into some of my sites and the colo company simply pulled the plugs (literally) due to Terms of Service violations (which of course weren't even done by me, but by the hackers using my servers to send out spam, and they wouldn't even let me appeal or anything).  Luckily I had a friend living in the city where my server was and had to have him drive to the facility, pick up my server and mail it to me from the U.S.

In any case, you are 100% right and there's no doubt who is _really_ to blame here, but there's a reality to it too, and the only thing I can think to say is to remember the saying "fool me once..."


----------

