Incident Updates (August 2021)

Incident Updates (August 2021)

Share this article

The contents of this blog followed the August 28, 2021 incident to provide live updates throughout the data recovery and restoration efforts. The final update was posted on September 13, 2021 as the situation shifted to a case-by-case basis. Any further communications have been sent to clients through their Client Area and/or email.

If your services are impacted, please refer to our Guide that will walk you through exactly what you can do right now depending on your situation. You may also refer to our FAQ for more details on the incident, or contact support through chat or ticket for assistance.

Final Update - September 13, 2021 - 5PM

Our large-scale recovery efforts have proven highly effective and we have been successful at restoring most impacted accounts through our courtesy backups.

Moving forward, we will be communicating our recovery assistance directly with clients through our support system on a case-by-case basis, and will continue to work with any impacted clients to help them resolve any remaining issues specific to their accounts. Additional communications will also be available on specific services in your Client Area.

In the meantime, if you are one of the impacted clients, please refer to our Guide that will walk you through exactly what you can do right now depending on which scenario best represents your situation, or contact support if you require personalized assistance.

We wish to thank our clients and partners and colleagues who have reached out to offer their support and encouragement. It kept us focused and motivated as we continued working tirelessly through this difficult time.

Current status per server

Last update: September 13, 2021 - 5PM

Update September 10, 2021 - 5PM

Data recovery process has been mostly completed and we have now recovered the majority of data from our courtesy backups on all impacted servers. This data should now be available directly on your original account, which you can access from your Client Area.

During our restoration effort, in order to avoid overwriting any live data, we used the following approach:

  1. If your original account was empty: we restored your content directly in your original account.
  2. If your original account already contained data: your backup should be available in your cPanel, under JetBackup. This gives you the option to choose exactly what to restore depending on your needs.

We have created this Help Center article that will walk you through exactly what you can do right now depending on which scenario best represents your situation. This guide includes detailed answers and step-by-step instructions for the most common problems following this restoration.

Potential of data corruption: Some of the data we recovered may have been corrupted. The level of data corruption we have seen so far is minimal. However, even a small level of data corruption might prevent a website from functioning fully.

You can refer to our corruption issues troubleshooting FAQ or contact our Support team for assistance. We have been successful at helping resolve several of these issues so far.

Our Support team will continue to be available to assist you, however we are currently experiencing a very large volume of communications that may end up causing delays.

We thank you for your continued patience and cooperation during this period.

Update September 9, 2021 - 3:45PM

Here’s an overview of where we’re at today...

Any remaining technical issues identified yesterday on Acadie and Bernard servers are now fully resolved. Websites, emails and databases should be working normally. Please contact support through chat or ticket if you have any issues.

Restoration is now mainly complete for Drummond, Bishop and Clark servers with some data corruption issues that may still prevent some websites from showing up online.

In order not to overwrite any live data, we refrained from restoring any account that already contained data. For these accounts, courtesy backups are now available in your cPanel so you can choose what to restore depending on where you are in your rebuilding efforts. REMINDER: Be very careful when performing a full account restore. This may overwrite any changes you've made since the backup date you select. As always, before initiating any restoration, we strongly recommend to first create a local backup of any live data if you've been working on your account. Check out this article to see how to manage and restore backups.

If your domain isn't currently pointing to your original account, you can use the ‘Preview Website’ tool in your Client Area to preview the state of your restored website.

Right now, if you are on servers Clark, Bishop or Drummond, your website is either working properly or it may be impacted by corrupted data.

About data corruption: The level of data corruption we have seen so far is minimal. However, even the smallest level of data corruption can prevent a website from functioning properly. The small subset of corrupted data is also scattered across many accounts, which makes it more challenging for us to find and fix. We continue to work case-by-case to resolve the varied ways the data corruption is manifesting itself and preventing certain websites from working.

For any data corruption issues, please contact our Support team through live chat or ticket. Our team has been successful at helping resolve the majority of issues on most websites. We are also building tools to help expedite this process.

Thank you again for your support and patience as we continue the restoration efforts round the clock.

Update September 8, 2021 - 1:30PM

There were a few minor hiccups overnight during our restoration effort, however we remain cautiously optimistic that we will have most accounts up and running by tomorrow morning. Here’s where we’re at:

For Acadie and Bernard servers, we’re currently working through some remaining technical issues for these servers and we will have an update as soon as we can.

The restoration process for ‘empty accounts’ on the Clark & Clark2 servers is almost done. Restoration for empty accounts on Drummond & Drummond2 servers is expected to be done by tonight.

Bishop & Bishop2 data recovery was completed with some minor data corruption. Restoration of empty accounts is also mostly complete at this point.

In order not to overwrite any live data, we refrained from restoring any account that contained live data. For these accounts, courtesy backups are now available in your Control Panel so you can choose what to restore depending on where you are in your rebuilding efforts. REMINDER: Be very careful when performing a full account restore. This may overwrite any changes you've made since the backup date you select. As always, before initiating any restoration, we strongly recommend to first create a local backup of any live data if you've been working on your account. Check out this article to see how to manage and restore backups.

Empty accounts: Accounts containing no data at time of restoration
Data corruption: Data corruption can manifest itself in many ways. e.g. you may see a broken plugin, a garbled page or other rendition issues. Our Support team has been successful at resolving most cases so far, and will continue working with you.

Your patience, understanding and support throughout this situation is what keeps us focused on our ultimate goal. We want to say a big THANK YOU (and one that comes straight from our hearts.)

Update September 7, 2021 - 1:30PM

The restoration process on Bishop is now nearly complete from our courtesy backups. The restoration process on Clark and Drummond is in progress and should be completed by tomorrow evening, barring any unforeseen circumstances.

A small part of the restored data appears to have some level of data corruption, but our Support team has been successful at resolving most cases so far. Please note that we have restored data from available courtesy backups only for accounts that were empty. For accounts that contained data (generally because you've re-uploaded your own content already), courtesy backups are now available directly in your Control Panel for you to restore your website using JetBackup. Please be very careful when restoring backups from your control panel as this may overwrite any changes you've made since the backup date. Always ensure you download a backup first.

Take note that our restoration efforts will NOT negatively impact the progress of those who are actively using their LifeBoat account.

Once the restorations are done, we will have a better understanding of the scope of the data corruption issue and can assist impacted clients with merging their LifeBoat account data, if needed.

Know that we will continue to work with you to resolve any issues that may arise now, or anytime in the near or distant future. Thank you once again for being patient and understanding. We’re almost there.

Update September 6, 2021 - 6:45PM

Good progress today. For Clark and Bishop servers, more and more accounts are starting to come back up as restoration progresses.

The restoration process for Drummond should start tonight or early tomorrow.

Restores are taking time, even if we’ve significantly increased our bandwidth and resources, to help speed up the process. We continue prioritizing integrity of data over expediency. We’re still cautiously optimistic that within the next few days, restoration will be complete with a high success rate.

As anticipated, we have witnessed minimal data corruption, and we’re relatively confident that we will be able to resolve related issues. Our Support team remains available if you need any assistance.

It’s important to note that clients with data on their live account should have access to their courtesy backups from before the incident in their control panel by tomorrow morning for Clark and Bishop, and later on this week for Drummond.

We're slowly getting closer to the finish line. Thank you again for your patience and understanding.

Update September 6, 2021 - 12:30PM

Things are looking quite good this morning for Clark and Bishop servers. The courtesy backups are now completely recovered. Restoration on these servers has now begun and we expect this process to be completed by late tomorrow. In the meantime, emails and websites will start coming back up online throughout the day as the restoration process advances.

For those who started rebuilding on their LifeBoat account, our restoration efforts will NOT affect your progress in any way. Also, you will ONLY be able to see your original account pop back online IF your website is not already live using your LifeBoat account.

The Drummond backups are continuing to recover today, we expect these to be done and ready to start restoration by tomorrow.

Please note: During our restoration efforts, we will avoid overwriting any live data. Once the courtesy backups are recovered, we will make them available for you in cPanel. This way you can choose which services to restore based on where you are in your rebuilding efforts.

Slowly but surely, we are getting there! We continue to prioritize data integrity throughout the entire restoration process. We have identified some data corruption issues and are devising strategies to handle such issues once all data has been restored. More details and updates on restoration progress will be provided later today.

Update September 5, 2021 - 9PM

We are currently working at resolving a kernel boot issue with Lifeboat service on Noah2. We hope to have all services back up on this server shortly.
UPDATE 9:35PM: Service to Noah2 is now resolved.

Update September 5, 2021 - 8PM

Good news, we have successfully recovered the courtesy backup data for all impacted accounts on the Rev server. Everything we could restore safely has been restored. We recommend you inspect your website/s and if everything is functioning as normal, you’re good to go! In order to avoid potentially overwriting live data, we made your courtesy backups available in cPanel, for you to restore your website using JetBackup. This way you can decide what’s the right move for you at this point. Friendly reminder: Make sure to choose a restore date prior to August 28. You can always contact support for any assistance.

We also made good progress and recovery is going well for Clark, Bishop and Drummond. We hope to begin restoration efforts on these servers by tomorrow morning. Nothing like a productive day to keep us motivated!
Thank you once again for your continued patience and understanding. Continue sending us your good vibes, it really makes all the difference.

Update September 5, 2021 - 12:30PM

The vast majority of Rev accounts are up and running. For clients still impacted, the Rev server is partially recovered, subsequent restoration is still progressing and is expected to be completed by tonight.

The situation we’re facing is still fluid, we continue making the best decisions based on how the situation develops. Yesterday, we encountered an issue during the recovery of Clark, Bishop and Drummond as some data corruption slowed our efforts. Our team worked their way through this issue. As such, recovery might take a little longer than anticipated. This is uncharted territory, and we continue to make every effort to speed up the recovery process.

We’ll have a much more accurate estimate by the end of day today, as to how long the remaining recovery efforts will take. Rest assured we are continuing to prioritize data integrity.

We appreciate your continued patience and support as we keep on working hard towards getting the remaining accounts up and running as fast as possible.

Update September 4, 2021 - 8PM

Please refer to the ‘Current status per server’ table above for updates on each impacted server.

Available Rev backups will likely be restored within 24 hours, and we are continuing to work diligently to advance our data recovery efforts on Clark, Drummond and Bishop.

We sincerely thank all our wonderful clients for the outpouring of support and encouragement. Your words have had an incredibly positive impact on us.

Update September 4, 2021 - 11AM

Note: You can now check your Client Area for up-to-date information concerning the latest developments on your account! Live updates will appear as notifications when logged in to your client area.

Here are the most recent developments: We now have full data recovery for servers: Acadie & Bernard. The entire restoration has been completed including websites, emails & databases. If you’re on either of these servers, please check your website and report any issues by opening a ticket for our support team.

The recovery for Rev is still in progress, most data is being recovered but we do have some data loss. High probability of data recovery for 95% of content within 24-48 hours.

Please note that we’re devising strategies to ensure that all accounts come up as quickly as possible. Be assured that we will continue our efforts throughout the weekend and beyond. For the clients who are in the midst of rebuilding, please note that restoration will not impact your efforts.

The encouragement and positive vibes we read online has been keeping us more motivated than ever!

Update September 3, 2021 - 4PM

Here are our most recent developments: Acadie & Bernard backups have now been recovered and we have commenced the restoration of accounts on these servers. There is a good chance that we may be able to have these accounts restored this evening.

Additionally, we now have access to 100% of lost datasets and are in the process of streaming them to healthy servers. Please note that some of these datasets WILL have some corruption, we anticipate this will affect approximately 2-5% of the data, but could be spread over many accounts. It's too early to tell how this corruption will impact any clients, but if all goes well, it should be minimal.

We continue to prioritize data integrity, and then expediency, to get accounts up and running as soon as possible.

Please note: We have confirmed this was not a ransomware attack and there is no indication that data of any kind was ever downloaded, exported, shared, or exposed.

Update September 3, 2021 - 11AM

Please view the ‘Current status per server’ list above for updates on each of the affected servers.

This morning, we are still streaming data from one of the impacted backup servers to a healthy server. We’re still cautiously optimistic that we will be able to continue recovering data from servers: Clark & Drummond, along with missing data from Rev. More updates will be provided on this front as our recovery efforts continue.

As stated last night, we should be able to know by this evening whether this same approach can be used to recover more data for servers: Acadie, Bernard and Bishop.

The situation we’re facing is very fluid and changes by the day (and even by the hour), we have been making the best decisions we can based on the information we have at any given time.

Whatever decision you make on your end, whether or not you choose to continue on a LifeBoat or wait in the hopes of having a recovered account; we will support you. Do not be concerned that you will make the wrong move in your own efforts. We can migrate your LifeBoat to your original account without issue once this situation has been resolved.

We also wanted to thank everyone who has been reaching out with words of encouragement and support. Our team has been thriving off these (and coffee).

Update September 2, 2021 - 7:45PM

We have some encouraging news to share.

We’ve successfully accessed data from one of the damaged backup servers and are currently streaming it to a healthy server. We are cautiously optimistic at this point that we will be able to recover most of the data from servers: Clark & Drummond, along with missing data from Rev.

Furthermore, we should be able to know by Friday evening whether this same approach can be used to recover more data for servers: Acadie, Bernard and Bishop. We are cautiously optimistic that we may be able to recover some of their data as well.

Earlier today, in a separate recovery effort, our team has successfully restored most files (excluding databases) and emails on servers Acadie and Bernard, which you already have access to now. However, tonight’s latest developments may indicate that we will be in a position to provide a more complete backup restoration for these accounts within the next few days.

Even though these are positive developments, there remains substantial work to be done, and the next 24 hours will be critical. It’s still too early to provide an ETA, but we will have a much better idea by tomorrow.

Keep sending us your positive vibes!

Update September 2, 2021 - 1:15PM

As we continue our recovery efforts we have created an FAQ resource that provides thorough answers to the most frequently asked questions our customer support team has been receiving so far.

Please refer to this FAQ before contacting our support team.

To reiterate, our Customer support has been experiencing a spike in chat and ticket requests and are doing their best to reply as quickly as possible. We want to apologize for the extra delay and thank you for your continued patience here.

It came to our attention that there may still be issues remaining with the previously recovered Peel server on some accounts. We’re currently working on a solution here.

Update September 1, 2021 - 10:30PM

Good evening, everyone. Today was a big day and a lot got done, with many more accounts coming online.

Tonight's update: Acadie and Bernard servers: file restoration (excluding databases) is ongoing and should be completed by early morning tomorrow. We believe we should be able to restore most emails on these servers.

Data recovery efforts on remaining servers are progressing on multiple fronts and we hope to get a better idea of what data can be salvaged in the coming days.

Update September 1, 2021 - 1:00PM

Here are the latest updates on the 6 of servers still impacted:

Clark, Drummond and Bishop servers: new empty accounts are now available in your Client Area, under your original service, to allow you to start rebuilding. If you are already using the LifeBoat account, you can continue using it without issues. In parallel, we are working with top disaster recovery teams to attempt to recuperate what we can but the progress is going to be slow and unlikely to result in a full recovery of data. You can use your new accounts knowing that they will not in any way hinder our recovery efforts.

Acadie and Bernard servers: we have secured files (but not databases) and are in the process of restoring them. They will be made available to you as possible (ETA: later today)

In addition, some data loss occurred for a small subset of resellers on server Rev. To help stabilize existing accounts, we are scheduling a migration of all data & accounts tonight and will be including data from this server in our recovery efforts. Resellers with missing data there are strongly encouraged to restore functionality from local backups.

Update September 1, 2021 - 11:45AM

Accounts on fully-recovered servers (Peel, Atwater and Beaubien) should be in working order now. There were problems with wrong dedicated IPs after we announced their full recovery yesterday, but this issue has since been addressed by our team overnight and everything should be working on these servers now. If you have any remaining issues, please reach out to customer support via live chat or ticket.

Please note: Customer support is experiencing a spike in chat and ticket requests and are doing their best to reply as quickly as possible. We will be providing an FAQ resource that will help to answer the large majority of questions that continue to pop up from our clients. This should help to reduce longer response times from our customer experience team. We, once again, want to apologize for the extra delay and thank you for your continued patience here.

Update August 31, 2021 - 7:30PM

We have now successfully completed 100% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now:
- Beaubien
- Peel
- Rachel
- Atwater

If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored.

Our efforts and attention has now shifted to servers with unrecoverable backups. Tonight, we'll finish activating replacement servers for the 5 we've lost, and by tomorrow morning your original hosting accounts will be once again available (although without data). For clients on Acadie & Bernard, we'll be automatically restoring the files that we managed to recover, and this should include all your emails but not databases. In the meanwhile, you can safely continue using your LifeBoat accounts.

For those hosted on servers Clark, Bishop and Drummond, we continue to work with the data recovery teams but these efforts are likely to last weeks, not days.

We'll provide an update tomorrow morning with next steps.

Update August 31, 2021 - 2:30PM

We have now successfully completed 99.9% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now:
- Beaubien
- Peel
- Rachel
- Atwater (99% done)

If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored.

Our efforts and attention is now shifting to servers with unrecoverable backups. The situation is still critical, but there are some (limited) positive developments. On 2 of the affected servers, specifically Acadie & Bernard, we were able to retrieve most of the files in users' accounts. This unfortunately excludes databases, but should include email content.

In the next 24-48 hours, we expect to make available fresh production servers (to replace the 5 we have lost) to which we will start streaming available data, as we find it. For the other servers, we're working with top data recovery teams to attempt to extract what we can, but the timeline for this is likely weeks, not days, and the results are far from certain.

We've also been receiving a lot of questions about the LifeBoat accounts that we've made available. These accounts are already available to you if you are one of those 5 servers where data recovery is unlikely. You can use these NOW to set up new emails and website content on your existing domain (follow these instructions), so you can keep business essential operations at least somewhat flowing. These accounts will remain active along your primary account (once it gets reactivated) so you'll be able to easily transfer data between them at a later date, if needed. The nameservers for the LifeBoats are ark1.whc.ca & ark2.whc.ca

More updates to follow as we have them.

Update August 31, 2021 - 11:00AM

Servers that have been fully or partially recovered

We have successfully recovered the following servers:
- Peel
- Rachel

Accounts on servers being restored now

The restoration process is continuing to advance and we hope to be able to fully recover all accounts on this server from backups by 3pm. Here’s the latest progress:
- Atwater is at 92%

Accounts on servers with damaged backups

For all customers on servers that have had both their local storage and their external backup storage heavily damaged, we are in the process of developing an informative Rescue Portal. This resource will provide you with all the information required to help you quickly and immediately start over in your new Lifeboat account.

Update August 30, 2021 - 10:00PM

As our team continues to work at restoring accounts, live updates are paused until tomorrow at 10AM, to better focus on the restoration efforts. We’ll be back tomorrow morning to give you an account of what has been accomplished overnight.

Servers that have been fully or partially recovered

The restoration process is advancing for the 3 servers where backups are available:
- Peel is at 96% (Should be complete by 2AM)
- Atwater is at 78%
- Rachel is at 85%

Update August 30, 2021 - 6:00PM

Servers that have been fully or partially recovered

We have successfully recovered the following servers:
- Beaubien
- Decarie
- Rev2
- Rev (some data loss has been reported here)

Accounts on servers with high likelihood of data recovery

The restoration process is advancing for the 3 servers where backups are available:
- Peel is at 82%
- Atwater is at 73%
- Rachel is at 76%

Accounts on servers with damaged backups

The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

Here is how you can move forward if your account was on one of those servers:

  • If you HAVE an adequate local backup: contact our support team and we will get you up and running on a new account ASAP
  • If you DON’T HAVE a local backup: You will need to start over from a bare, empty account. To this end, we have activated new, free hosting accounts for each impacted domain, called LifeBoat. They are available in your Client Area now and are intended to serve as a quick, free and immediate way for you to start over. These LifeBoat accounts will remain free of charge until at least January 1, 2022. To get started and learn how to use your new LifeBoat account, please follow these steps.

Update August 30, 2021 - 3:00PM

The restoration process is advancing for the 4 servers where backups are available:
- Beaubien is at 99.9%
- Peel is at 73.62%
- Atwater is at 71.43%
- Rachel is at 71.29%

There is no update at this time for clients on servers with damaged backups. We continue to recommend that you use the new LifeBoat account to help continue your business operations. We will make additional resources available soon to you to help with the rebuild process.

Update August 30, 2021 - 9:00AM

The restoration process is advancing for the 4 servers where backups are available:
- Beaubien is at 88%
- Peel is at 65%
- Atwater is at 65%
- Rachel is at 64%

For clients on servers with damaged backups, we recommend you initiate your disaster recovery plan ASAP. We have created hosting accounts in your Client Area called "LifeBoats" to help you get up and running while we explore long-term data recovery options. The impacted servers are:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

Update August 29, 2021 - 11:00PM

Accounts on servers with damaged backups

With our initial data recovery process unsuccessful, we’re now recommending that all affected clients that have experienced data loss consider using a new, temporary hosting account that’s already created in their Client Area called LifeBoat.

To use this account, click into it to create your emails and upload website data just as you would on your regular hosting account.

Once this is done, just link your domain to this account, update your domain’s DNS to:

ark1.whc.ca
ark2.whc.ca

Once the DNS changes propagate (can take several hours) your new account can be used to start accepting and sending new email and upload new website copy quickly. These accounts will remain free of charge until at least January 1, 2022 and are intended as a stopgap measure until a more permanent solution is found.

Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:

- Beaubien is at 62%
- Peel is at 45%
- Atwater is at 45%
- Rachel is at 31%

Update August 29, 2021 - 5:00PM

As our team continues to work to restore accounts, please note that live updates are paused until tomorrow morning at 10AM so we can again focus on the restoration efforts. We’ll then be able to give an account of what has been accomplished overnight.

Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:
- Beaubien is at 55%
- Peel is at 35%
- Atwater is at 30%
- Rachel is at 10%

Accounts on servers with damaged backups

The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

Our initial attempt to repair the data on our backup servers has failed and at this point the likelihood of successfully restoring account data from these servers is very low.

As a result, we are working with data recovery experts to attempt to restore data from the source server, but this process is extremely long and tedious, potentially requiring file-by-file analysis for systems with millions of files and may require months to complete.

Here is how you can move forward if your account was on one of those servers:

  • If you have a local backup
    If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
  • If you don’t have a local backup: introducing “Lifeboat” accounts
    To facilitate continued business operations for our clients during this major outage and until the situation is resolved, we will be activating temporary hosting accounts for all clients hosted on Clark, Drummond, Acadie, Bernard and Bishop servers. You will be able to log in to this new account in order to recreate your emails and upload website content, without hindering ongoing recovery efforts. A simple nameserver change will be required and step-by-step instructions will be provided. These accounts are expected to be available before Monday morning.
For further assistance

Please use our Live Chat or tickets for help requests. Phone lines are temporarily closed to better support restoration efforts. We thank you for your continued patience and understanding.

Update August 29, 2021 - 11:00AM

Thank you everyone for your patience and support during what has been a very trying 24 hours. Our team has been working tirelessly to restore service to affected clients and have had some success but the news isn’t all good, so here is where we stand.

Servers that have been fully or partially recovered

We have successfully recovered the following servers:
- Decarie
- Rev2
- Rev (some data loss has been reported here)

Accounts on servers with high likelihood of data recovery

We have 4 systems whose external backup servers have not been impacted, and these are actively being used to stream account restorations right now. We expect to be able to fully restore all backed up accounts by Monday evening and hopefully sooner. Here is where they are in the restore process:
- Beaubien (50%)
- Peel (30%)
- Atwater (20%)
- Rachel (1%, restore has just been started)

A full restore process for a system of this size generally takes 24 hours. As soon as an account is restored, its website and email should become available so for most users on these machines you should have your services restored before the end of the day Sunday.

Accounts on servers with low likelihood of data recovery

The following 5 systems have had their external backup servers also partially destroyed, and we have been as of yet unsuccessful at recovering their data:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

For clients on these machines, we are pursuing 3 recovery strategies in parallel:

  1. Re-upload your content from your own backup
    If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
  2. Partial data recovery on production servers
    We have had some success in recovering hosting accounts data files (excluding databases) for 2 of the affected servers (Acadie & Bernard for now) and may be able to perform a partial restoration of these accounts within the next 48-72 hours. This process is slow and tedious and unlikely to complete before Tuesday.
  3. Manual data recovery
    We’ve contracted an external firm to assist us with data recovery efforts on our backup servers and they started their work on Saturday night. Manual data recovery is a tedious and slow process, but we expect to get a report on what can be salvaged before day’s end. This approach will likely take the longest to result in recovered data, with a timeline as long as 1-2 weeks.

We will be posting more information along with a post-mortem once the immediate operational issues have been addressed.

We have mobilized our full team to assist in this effort and we currently have dozens of system administrators, support technicians and data recovery engineers assisting with the recovery effort since yesterday. We understand this incident may seriously impact your business and are doing everything in our power to mitigate the damage.

More information will follow as we have it.

Update August 29, 2021 - 10:00AM

Thank you once again for your patience. We continued to work overnight on restoring accounts.

Here are the updates for each of our restored servers so far:
- Rev2 server has been fully recovered and should be functioning normally
- Decarie server has been fully recovered and should be functioning normally
- Rev server was recovered, however, some issues remain

Here are the updates for servers currently in the restoration process:
- Rachel's restoration process should begin within the next hour
- Peel restoration is approaching 30% completion.
- Beaubien restoration is approaching 50% completion.
- Atwater restoration is approaching 20% completion.

Services on the above servers should be fully functional within the next 12-24 hours.

Please note: We will be posting another update at 11AM for the more severely impacted servers.

Update August 28, 2021 - 9:00PM

As our team continues to work overnight to restore accounts, please note that live updates will be paused until tomorrow morning at 10AM so we can focus on the restoration efforts. We’ll then be able to give a more detailed account of what has been accomplished.

If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account.

While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority.

Servers with more severe data loss are shown below:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

Update August 28, 2021 - 7:50PM

We have now proceeded to restore the servers:
- Rev
- Decarie
- Rev2

The following servers are being restored from disaster recovery backups and accounts are gradually coming back up as they restore (full process will exceed 24 hours):
- Beaubien
- Peel
- Rachel
- Atwater

The following servers have suffered from more severe data loss and we are investigating advanced data recovery steps:
- Clark
- Drummond
- Acadie
- Bernard
- Bishop

If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account.

While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority.

Update August 28, 2021 - 5:10PM

We have now proceeded to restore the servers:
- Decarie
- Rev2

The following servers are being restored from disaster recovery backups:
- Beaubien
- Peel
- Rachel
- Atwater

We are still investigating the situation for the following servers and do not yet have an update at this time:
- Clark
- Drummond
- Rev
- Acadie
- Bernard
- Bishop

Update August 28, 2021 - 3PM

We have begun the process for disaster recovery from external backups for the following servers:
- Beaubien
- Peel
- Rachel
- Atwater

Considering this is a full system backup restore, restoration can last up to 24 hours before all the data is available again. We will provide updates as they become available.

We are still investigating the situation for the following servers and do not yet have an update at this time:
- Clark
- Decarie
- Drummond
- Rev
- Acadie
- Bernard
- Bishop

Update August 28, 2021 - 1:40PM

We were able to successfully restore service to Rev2 server, which is back up and running. We are advancing in our efforts to restore other systems.

Update August 28, 2021 - 1:25PM

Systems Beaubien & Peel have confirmed working backups and we are proceeding with full backup restore. The following systems are still down:

- Clark
- Drummond
- Rev
- Rev2
- Acadie
- Beaubien
- Bernard
- Bishop
- Atwater
- Decarie
- Rachel

We are continuing to investigate recovery strategy for these systems and will update soon.

Update August 28, 2021 - 1PM

We can confirm that multiple systems are impacted by a major incident right now. We are attempting to recover source data and restore service functionality and will post updates as soon as they become available.

We sincerely apologize for any inconvenience caused and appreciate your patience as we continue to actively work on understanding and resolving this incident.

Please note: Contacting support at this time will not result in any further updates or resolutions, and may add to delays.

As details become available, all updates will be communicated LIVE here.

Back
Share this article Facebook Twitter LinkedIn By Email
About the author: Emil Falcon

Emil is WHC's Founder and CEO. When he’s not busy obsessing about things such as optimizing workflows or enhancing system performance, security and reliability, he’s spearheading initiatives that can help entrepreneurs and businesses succeed across Canada.

See all articles from this author Interested in writing for Web Hosting Canada?
Web Hosting Canada manages fast and reliable online infrastructure with 24/7 support. Learn more about WHC