Major Incident: What happened?

It’s been a tough weekend here at WHC and by this, I include our clients. I want to start by thanking all the team for coming together and working through the problem constructively and with tremendous heart and energy.

Here’s the situation.

Based on our investigation to date, the morning of August 28 at approximately 6AM, an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers.

Within only hours our incident response team had identified the issue and disabled access to the source account, preventing any further damage. The environment was secured, the individual fully locked out, and our disaster recovery plan immediately kicked into action but damage was already done.

The tally of the incident, however, was and still is important: a few major systems, including some production servers and some backup servers were damaged, with a large number of web hosting and reseller hosting accounts affected, resulting in possible permanent data loss.

After nearly 2 days of tedious work and a combination of external datacenter backup restores and system-level storage rebuilding, our team was able to successfully recover (or is in the process of recovering) over 50% of those lost accounts. We can confirm that Cloud, Dedicated, Weebly and Managed WordPress accounts were largely unaffected.

Unfortunately, at the moment, I can tell you that several production servers and their respective backup servers are still in an unrecoverable state and the data recovery experts assisting us believe that the recovery potential is very low. As such, the focus for these accounts has shifted from data recovery to starting fresh on new, clean accounts. In parallel we will continue to attempt to recover any data we can.

For clients impacted by this incident and for which we are unable to recover backups:

• If you have an adequate local backup: contact our support team and we will get you up and running on a new account ASAP
• If you do not have a local backup: You will need to start over from a bare, empty account. To this end, we have activated new, free hosting accounts for each impacted domain, called LifeBoat. They are available in your Client Area now and are intended to serve as a quick, free and immediate way for you to start over. These LifeBoat accounts will remain free of charge until at least January 1, 2022.

On behalf of WHC, I would like to extend our sincere apologies to those affected by this unfortunate situation. With the cooperation of those particular clients affected by the incident, we believe that we can greatly minimise the consequences stemming from this involuntary incident.

We remain committed as ever to providing you with quality and affordable hosting solutions. We understand and regret the impact that this incident may have on your business and operations.

We are also grateful and moved by the outpouring of support we have received as we continue working to tackle this issue.

Sincerely,

Emil Falcon
CEO at Web Hosting Canada