Archives

What we’ve done & how we’re moving forward

WHC News & Events
Dear valued clients, For those of you impacted by the event of August 28, I want to reach out again to convey our heartfelt apologies. To those who worked with us to restore data and operations, we thank you. And to the large majority of clients who were not impacted by the incident, I wanted to provide further transparency and reassurance. I am proud to report that nearly all impacted accounts have been recovered or restored from courtesy backups with the vast majority back online. If you are still having issues, please contact our support team for rapid assistance. Here’s an update of what we’ve done and how we’re moving forward.

What we’ve done

  • Identified the root cause and limited its impact
    Immediately upon becoming aware of the issue, steps were taken to contain the incident and an internal verification confirmed our systems were secure. There is no indication that data of any kind was ever downloaded, exported, shared, or exposed.
  • Restored damaged accounts from courtesy backups
    We initiated restoration of impacted accounts from available working courtesy backups, and most of these accounts were back online and available within 48 hours of the incident.
  • Successfully recovered data from damaged backup servers
    Undoubtedly the longest and hardest part of our recovery efforts involved rebuilding damaged storage arrays. This process involved working with file system experts and included building file system patches to allow successful transfers of over 98% of data. In parallel, we explored alternate recovery options and were successful as a result to restore access to many more accounts even faster. As soon as we had sufficient confidence that the data was recoverable, we communicated the encouraging news to clients.
  • Provided courtesy LifeBoat accounts
    We quickly provided courtesy hosting accounts for clients whose backups were damaged, enabling them to recover their sites from their own local backups without impacting our recovery efforts. These accounts will continue to be available for the near future.
  • Scaled up our support teams and optimized our processes
    We rapidly scaled up our support and systems teams’ availability to help assist clients in their recovery, focusing our efforts where they would be most effective.
  • Built automated tools to help fix websites even faster
    As we began to better understand the scope of the data corruption affecting some accounts, we built and used tools that helped detect and fix problematic sites.

How we’re moving forward

As we return to normal operations, I understand some of you may have questions and will want to be reassured. Our clients expect maximum uptime, excellent customer service, and solid performance and security, and we want to continue delivering exactly this. Cybersecurity is an evolving landscape and WHC is committed to staying ahead of new, emerging threats. We have learned from this incident and we are much stronger because of it. These are our commitments to you, as a valued client/partner. We will continue to:
  • Further strengthen security
    We have already invested heavily in security and will continue to invest significant time and energy to confront and neutralize new and emerging threats. While the nature of technology prevents anyone from guaranteeing 100% security, we will deploy all reasonable efforts to continue to help ensure a safe, secure environment for your data.
  • Better inform and communicate with you
    We will continue and step up efforts to educate our clients on best practices surrounding data management and protection, including disaster recovery, backups, and online security, through our blog, newsletters and knowledge base. We will also provide an improved live status portal to provide greater insight and transparency.
  • Better support you
    We will continue to grow our team to be able to better serve your needs. Our team continues to be here 24/7 to support you.
  • Enhance our hosting platform
    We are improving our infrastructure to better serve the needs of Canadian small businesses. In particular, we plan on rolling out specific reporting tools to give web hosting users a better understanding of issues and errors on their websites, as well as overall performance.
  • Stand behind our uptime guarantee
    We honour our commitments and stand by our 99.9% uptime guarantee. If you have been impacted by this incident and have not already done so, you may file a claim from your secure Client Area here.
  • Provide LifeBoat and Migration assistance to impacted clients
    For those still requiring assistance getting back on their feet, we will be with you every step of the way. This includes our LifeBoat service, available free of charge to those needing it until the end of the calendar year as well as our dedicated support team to assist you with merging account data back into your primary account.
This has been a tough few weeks for both impacted clients and the entire WHC team. However, the outcome has been a positive one and our systems are fully operational. As CEO of Web Hosting Canada, let me also reiterate our core commitments to you:
  • We will continue to diligently and wholeheartedly help Canadian small businesses thrive online
  • We will continue to invest in and improve our systems and services
  • We will continue to deliver our services ethically, honestly and transparently
...Because this is what we do and this is who we are. To our clients, team and partners who were extraordinary through this whole process: we thank you for your patience, understanding and support. We look forward to continuing our online journey together. Sincerely, Emil Falcon & The WHC team

Incident Updates (August 2021)

WHC News & Events
The contents of this blog followed the August 28, 2021 incident to provide live updates throughout the data recovery and restoration efforts. The final update was posted on September 13, 2021 as the situation shifted to a case-by-case basis. Any further communications have been sent to clients through their Client Area and/or email.
If your services are impacted, please refer to our Guide that will walk you through exactly what you can do right now depending on your situation. You may also refer to our FAQ for more details on the incident, or contact support through chat or ticket for assistance.

Final Update - September 13, 2021 - 5PM

Our large-scale recovery efforts have proven highly effective and we have been successful at restoring most impacted accounts through our courtesy backups. Moving forward, we will be communicating our recovery assistance directly with clients through our support system on a case-by-case basis, and will continue to work with any impacted clients to help them resolve any remaining issues specific to their accounts. Additional communications will also be available on specific services in your Client Area. In the meantime, if you are one of the impacted clients, please refer to our Guide that will walk you through exactly what you can do right now depending on which scenario best represents your situation, or contact support if you require personalized assistance. We wish to thank our clients and partners and colleagues who have reached out to offer their support and encouragement. It kept us focused and motivated as we continued working tirelessly through this difficult time.

Current status per server

Last update: September 13, 2021 - 5PM

Update September 10, 2021 - 5PM

Data recovery process has been mostly completed and we have now recovered the majority of data from our courtesy backups on all impacted servers. This data should now be available directly on your original account, which you can access from your Client Area. During our restoration effort, in order to avoid overwriting any live data, we used the following approach:
  1. If your original account was empty: we restored your content directly in your original account.
  2. If your original account already contained data: your backup should be available in your cPanel, under JetBackup. This gives you the option to choose exactly what to restore depending on your needs.
We have created this Help Center article that will walk you through exactly what you can do right now depending on which scenario best represents your situation. This guide includes detailed answers and step-by-step instructions for the most common problems following this restoration. Potential of data corruption: Some of the data we recovered may have been corrupted. The level of data corruption we have seen so far is minimal. However, even a small level of data corruption might prevent a website from functioning fully. You can refer to our corruption issues troubleshooting FAQ or contact our Support team for assistance. We have been successful at helping resolve several of these issues so far. Our Support team will continue to be available to assist you, however we are currently experiencing a very large volume of communications that may end up causing delays. We thank you for your continued patience and cooperation during this period.

Update September 9, 2021 - 3:45PM

Here’s an overview of where we’re at today... Any remaining technical issues identified yesterday on Acadie and Bernard servers are now fully resolved. Websites, emails and databases should be working normally. Please contact support through chat or ticket if you have any issues. Restoration is now mainly complete for Drummond, Bishop and Clark servers with some data corruption issues that may still prevent some websites from showing up online. In order not to overwrite any live data, we refrained from restoring any account that already contained data. For these accounts, courtesy backups are now available in your cPanel so you can choose what to restore depending on where you are in your rebuilding efforts. REMINDER: Be very careful when performing a full account restore. This may overwrite any changes you've made since the backup date you select. As always, before initiating any restoration, we strongly recommend to first create a local backup of any live data if you've been working on your account. Check out this article to see how to manage and restore backups. If your domain isn't currently pointing to your original account, you can use the ‘Preview Website’ tool in your Client Area to preview the state of your restored website. Right now, if you are on servers Clark, Bishop or Drummond, your website is either working properly or it may be impacted by corrupted data. About data corruption: The level of data corruption we have seen so far is minimal. However, even the smallest level of data corruption can prevent a website from functioning properly. The small subset of corrupted data is also scattered across many accounts, which makes it more challenging for us to find and fix. We continue to work case-by-case to resolve the varied ways the data corruption is manifesting itself and preventing certain websites from working. For any data corruption issues, please contact our Support team through live chat or ticket. Our team has been successful at helping resolve the majority of issues on most websites. We are also building tools to help expedite this process. Thank you again for your support and patience as we continue the restoration efforts round the clock.

Update September 8, 2021 - 1:30PM

There were a few minor hiccups overnight during our restoration effort, however we remain cautiously optimistic that we will have most accounts up and running by tomorrow morning. Here’s where we’re at: For Acadie and Bernard servers, we’re currently working through some remaining technical issues for these servers and we will have an update as soon as we can. The restoration process for ‘empty accounts’ on the Clark & Clark2 servers is almost done. Restoration for empty accounts on Drummond & Drummond2 servers is expected to be done by tonight. Bishop & Bishop2 data recovery was completed with some minor data corruption. Restoration of empty accounts is also mostly complete at this point. In order not to overwrite any live data, we refrained from restoring any account that contained live data. For these accounts, courtesy backups are now available in your Control Panel so you can choose what to restore depending on where you are in your rebuilding efforts. REMINDER: Be very careful when performing a full account restore. This may overwrite any changes you've made since the backup date you select. As always, before initiating any restoration, we strongly recommend to first create a local backup of any live data if you've been working on your account. Check out this article to see how to manage and restore backups. Empty accounts: Accounts containing no data at time of restoration Data corruption: Data corruption can manifest itself in many ways. e.g. you may see a broken plugin, a garbled page or other rendition issues. Our Support team has been successful at resolving most cases so far, and will continue working with you. Your patience, understanding and support throughout this situation is what keeps us focused on our ultimate goal. We want to say a big THANK YOU (and one that comes straight from our hearts.)

Update September 7, 2021 - 1:30PM

The restoration process on Bishop is now nearly complete from our courtesy backups. The restoration process on Clark and Drummond is in progress and should be completed by tomorrow evening, barring any unforeseen circumstances. A small part of the restored data appears to have some level of data corruption, but our Support team has been successful at resolving most cases so far. Please note that we have restored data from available courtesy backups only for accounts that were empty. For accounts that contained data (generally because you've re-uploaded your own content already), courtesy backups are now available directly in your Control Panel for you to restore your website using JetBackup. Please be very careful when restoring backups from your control panel as this may overwrite any changes you've made since the backup date. Always ensure you download a backup first. Take note that our restoration efforts will NOT negatively impact the progress of those who are actively using their LifeBoat account. Once the restorations are done, we will have a better understanding of the scope of the data corruption issue and can assist impacted clients with merging their LifeBoat account data, if needed. Know that we will continue to work with you to resolve any issues that may arise now, or anytime in the near or distant future. Thank you once again for being patient and understanding. We’re almost there.

Update September 6, 2021 - 6:45PM

Good progress today. For Clark and Bishop servers, more and more accounts are starting to come back up as restoration progresses. The restoration process for Drummond should start tonight or early tomorrow. Restores are taking time, even if we’ve significantly increased our bandwidth and resources, to help speed up the process. We continue prioritizing integrity of data over expediency. We’re still cautiously optimistic that within the next few days, restoration will be complete with a high success rate. As anticipated, we have witnessed minimal data corruption, and we’re relatively confident that we will be able to resolve related issues. Our Support team remains available if you need any assistance. It’s important to note that clients with data on their live account should have access to their courtesy backups from before the incident in their control panel by tomorrow morning for Clark and Bishop, and later on this week for Drummond. We're slowly getting closer to the finish line. Thank you again for your patience and understanding.

Update September 6, 2021 - 12:30PM

Things are looking quite good this morning for Clark and Bishop servers. The courtesy backups are now completely recovered. Restoration on these servers has now begun and we expect this process to be completed by late tomorrow. In the meantime, emails and websites will start coming back up online throughout the day as the restoration process advances. For those who started rebuilding on their LifeBoat account, our restoration efforts will NOT affect your progress in any way. Also, you will ONLY be able to see your original account pop back online IF your website is not already live using your LifeBoat account. The Drummond backups are continuing to recover today, we expect these to be done and ready to start restoration by tomorrow. Please note: During our restoration efforts, we will avoid overwriting any live data. Once the courtesy backups are recovered, we will make them available for you in cPanel. This way you can choose which services to restore based on where you are in your rebuilding efforts. Slowly but surely, we are getting there! We continue to prioritize data integrity throughout the entire restoration process. We have identified some data corruption issues and are devising strategies to handle such issues once all data has been restored. More details and updates on restoration progress will be provided later today.

Update September 5, 2021 - 9PM

We are currently working at resolving a kernel boot issue with Lifeboat service on Noah2. We hope to have all services back up on this server shortly. UPDATE 9:35PM: Service to Noah2 is now resolved.

Update September 5, 2021 - 8PM

Good news, we have successfully recovered the courtesy backup data for all impacted accounts on the Rev server. Everything we could restore safely has been restored. We recommend you inspect your website/s and if everything is functioning as normal, you’re good to go! In order to avoid potentially overwriting live data, we made your courtesy backups available in cPanel, for you to restore your website using JetBackup. This way you can decide what’s the right move for you at this point. Friendly reminder: Make sure to choose a restore date prior to August 28. You can always contact support for any assistance. We also made good progress and recovery is going well for Clark, Bishop and Drummond. We hope to begin restoration efforts on these servers by tomorrow morning. Nothing like a productive day to keep us motivated! Thank you once again for your continued patience and understanding. Continue sending us your good vibes, it really makes all the difference.

Update September 5, 2021 - 12:30PM

The vast majority of Rev accounts are up and running. For clients still impacted, the Rev server is partially recovered, subsequent restoration is still progressing and is expected to be completed by tonight. The situation we’re facing is still fluid, we continue making the best decisions based on how the situation develops. Yesterday, we encountered an issue during the recovery of Clark, Bishop and Drummond as some data corruption slowed our efforts. Our team worked their way through this issue. As such, recovery might take a little longer than anticipated. This is uncharted territory, and we continue to make every effort to speed up the recovery process. We’ll have a much more accurate estimate by the end of day today, as to how long the remaining recovery efforts will take. Rest assured we are continuing to prioritize data integrity. We appreciate your continued patience and support as we keep on working hard towards getting the remaining accounts up and running as fast as possible.

Update September 4, 2021 - 8PM

Please refer to the ‘Current status per server’ table above for updates on each impacted server. Available Rev backups will likely be restored within 24 hours, and we are continuing to work diligently to advance our data recovery efforts on Clark, Drummond and Bishop. We sincerely thank all our wonderful clients for the outpouring of support and encouragement. Your words have had an incredibly positive impact on us.

Update September 4, 2021 - 11AM

Note: You can now check your Client Area for up-to-date information concerning the latest developments on your account! Live updates will appear as notifications when logged in to your client area. Here are the most recent developments: We now have full data recovery for servers: Acadie & Bernard. The entire restoration has been completed including websites, emails & databases. If you’re on either of these servers, please check your website and report any issues by opening a ticket for our support team. The recovery for Rev is still in progress, most data is being recovered but we do have some data loss. High probability of data recovery for 95% of content within 24-48 hours. Please note that we’re devising strategies to ensure that all accounts come up as quickly as possible. Be assured that we will continue our efforts throughout the weekend and beyond. For the clients who are in the midst of rebuilding, please note that restoration will not impact your efforts. The encouragement and positive vibes we read online has been keeping us more motivated than ever!

Update September 3, 2021 - 4PM

Here are our most recent developments: Acadie & Bernard backups have now been recovered and we have commenced the restoration of accounts on these servers. There is a good chance that we may be able to have these accounts restored this evening. Additionally, we now have access to 100% of lost datasets and are in the process of streaming them to healthy servers. Please note that some of these datasets WILL have some corruption, we anticipate this will affect approximately 2-5% of the data, but could be spread over many accounts. It's too early to tell how this corruption will impact any clients, but if all goes well, it should be minimal. We continue to prioritize data integrity, and then expediency, to get accounts up and running as soon as possible. Please note: We have confirmed this was not a ransomware attack and there is no indication that data of any kind was ever downloaded, exported, shared, or exposed.

Update September 3, 2021 - 11AM

Please view the ‘Current status per server’ list above for updates on each of the affected servers. This morning, we are still streaming data from one of the impacted backup servers to a healthy server. We’re still cautiously optimistic that we will be able to continue recovering data from servers: Clark & Drummond, along with missing data from Rev. More updates will be provided on this front as our recovery efforts continue. As stated last night, we should be able to know by this evening whether this same approach can be used to recover more data for servers: Acadie, Bernard and Bishop. The situation we’re facing is very fluid and changes by the day (and even by the hour), we have been making the best decisions we can based on the information we have at any given time. Whatever decision you make on your end, whether or not you choose to continue on a LifeBoat or wait in the hopes of having a recovered account; we will support you. Do not be concerned that you will make the wrong move in your own efforts. We can migrate your LifeBoat to your original account without issue once this situation has been resolved. We also wanted to thank everyone who has been reaching out with words of encouragement and support. Our team has been thriving off these (and coffee).

Update September 2, 2021 - 7:45PM

We have some encouraging news to share. We’ve successfully accessed data from one of the damaged backup servers and are currently streaming it to a healthy server. We are cautiously optimistic at this point that we will be able to recover most of the data from servers: Clark & Drummond, along with missing data from Rev. Furthermore, we should be able to know by Friday evening whether this same approach can be used to recover more data for servers: Acadie, Bernard and Bishop. We are cautiously optimistic that we may be able to recover some of their data as well. Earlier today, in a separate recovery effort, our team has successfully restored most files (excluding databases) and emails on servers Acadie and Bernard, which you already have access to now. However, tonight’s latest developments may indicate that we will be in a position to provide a more complete backup restoration for these accounts within the next few days. Even though these are positive developments, there remains substantial work to be done, and the next 24 hours will be critical. It’s still too early to provide an ETA, but we will have a much better idea by tomorrow. Keep sending us your positive vibes!

Update September 2, 2021 - 1:15PM

As we continue our recovery efforts we have created an FAQ resource that provides thorough answers to the most frequently asked questions our customer support team has been receiving so far. Please refer to this FAQ before contacting our support team. To reiterate, our Customer support has been experiencing a spike in chat and ticket requests and are doing their best to reply as quickly as possible. We want to apologize for the extra delay and thank you for your continued patience here. It came to our attention that there may still be issues remaining with the previously recovered Peel server on some accounts. We’re currently working on a solution here.

Update September 1, 2021 - 10:30PM

Good evening, everyone. Today was a big day and a lot got done, with many more accounts coming online. Tonight's update: Acadie and Bernard servers: file restoration (excluding databases) is ongoing and should be completed by early morning tomorrow. We believe we should be able to restore most emails on these servers. Data recovery efforts on remaining servers are progressing on multiple fronts and we hope to get a better idea of what data can be salvaged in the coming days.

Update September 1, 2021 - 1:00PM

Here are the latest updates on the 6 of servers still impacted: Clark, Drummond and Bishop servers: new empty accounts are now available in your Client Area, under your original service, to allow you to start rebuilding. If you are already using the LifeBoat account, you can continue using it without issues. In parallel, we are working with top disaster recovery teams to attempt to recuperate what we can but the progress is going to be slow and unlikely to result in a full recovery of data. You can use your new accounts knowing that they will not in any way hinder our recovery efforts. Acadie and Bernard servers: we have secured files (but not databases) and are in the process of restoring them. They will be made available to you as possible (ETA: later today) In addition, some data loss occurred for a small subset of resellers on server Rev. To help stabilize existing accounts, we are scheduling a migration of all data & accounts tonight and will be including data from this server in our recovery efforts. Resellers with missing data there are strongly encouraged to restore functionality from local backups.

Update September 1, 2021 - 11:45AM

Accounts on fully-recovered servers (Peel, Atwater and Beaubien) should be in working order now. There were problems with wrong dedicated IPs after we announced their full recovery yesterday, but this issue has since been addressed by our team overnight and everything should be working on these servers now. If you have any remaining issues, please reach out to customer support via live chat or ticket. Please note: Customer support is experiencing a spike in chat and ticket requests and are doing their best to reply as quickly as possible. We will be providing an FAQ resource that will help to answer the large majority of questions that continue to pop up from our clients. This should help to reduce longer response times from our customer experience team. We, once again, want to apologize for the extra delay and thank you for your continued patience here.

Update August 31, 2021 - 7:30PM

We have now successfully completed 100% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now: - Beaubien - Peel - Rachel - Atwater If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored. Our efforts and attention has now shifted to servers with unrecoverable backups. Tonight, we'll finish activating replacement servers for the 5 we've lost, and by tomorrow morning your original hosting accounts will be once again available (although without data). For clients on Acadie & Bernard, we'll be automatically restoring the files that we managed to recover, and this should include all your emails but not databases. In the meanwhile, you can safely continue using your LifeBoat accounts. For those hosted on servers Clark, Bishop and Drummond, we continue to work with the data recovery teams but these efforts are likely to last weeks, not days. We'll provide an update tomorrow morning with next steps.

Update August 31, 2021 - 2:30PM

We have now successfully completed 99.9% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now: - Beaubien - Peel - Rachel - Atwater (99% done) If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored. Our efforts and attention is now shifting to servers with unrecoverable backups. The situation is still critical, but there are some (limited) positive developments. On 2 of the affected servers, specifically Acadie & Bernard, we were able to retrieve most of the files in users' accounts. This unfortunately excludes databases, but should include email content. In the next 24-48 hours, we expect to make available fresh production servers (to replace the 5 we have lost) to which we will start streaming available data, as we find it. For the other servers, we're working with top data recovery teams to attempt to extract what we can, but the timeline for this is likely weeks, not days, and the results are far from certain. We've also been receiving a lot of questions about the LifeBoat accounts that we've made available. These accounts are already available to you if you are one of those 5 servers where data recovery is unlikely. You can use these NOW to set up new emails and website content on your existing domain (follow these instructions), so you can keep business essential operations at least somewhat flowing. These accounts will remain active along your primary account (once it gets reactivated) so you'll be able to easily transfer data between them at a later date, if needed. The nameservers for the LifeBoats are ark1.whc.ca & ark2.whc.ca More updates to follow as we have them.

Update August 31, 2021 - 11:00AM

Servers that have been fully or partially recovered
We have successfully recovered the following servers: - Peel - Rachel
Accounts on servers being restored now
The restoration process is continuing to advance and we hope to be able to fully recover all accounts on this server from backups by 3pm. Here’s the latest progress: - Atwater is at 92%
Accounts on servers with damaged backups
For all customers on servers that have had both their local storage and their external backup storage heavily damaged, we are in the process of developing an informative Rescue Portal. This resource will provide you with all the information required to help you quickly and immediately start over in your new Lifeboat account.

Update August 30, 2021 - 10:00PM

As our team continues to work at restoring accounts, live updates are paused until tomorrow at 10AM, to better focus on the restoration efforts. We’ll be back tomorrow morning to give you an account of what has been accomplished overnight.
Servers that have been fully or partially recovered
The restoration process is advancing for the 3 servers where backups are available: - Peel is at 96% (Should be complete by 2AM) - Atwater is at 78% - Rachel is at 85%

Update August 30, 2021 - 6:00PM

Servers that have been fully or partially recovered
We have successfully recovered the following servers: - Beaubien - Decarie - Rev2 - Rev (some data loss has been reported here)
Accounts on servers with high likelihood of data recovery
The restoration process is advancing for the 3 servers where backups are available: - Peel is at 82% - Atwater is at 73% - Rachel is at 76%
Accounts on servers with damaged backups
The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult: - Clark - Drummond - Acadie - Bernard - Bishop Here is how you can move forward if your account was on one of those servers:
  • If you HAVE an adequate local backup: contact our support team and we will get you up and running on a new account ASAP
  • If you DON’T HAVE a local backup: You will need to start over from a bare, empty account. To this end, we have activated new, free hosting accounts for each impacted domain, called LifeBoat. They are available in your Client Area now and are intended to serve as a quick, free and immediate way for you to start over. These LifeBoat accounts will remain free of charge until at least January 1, 2022. To get started and learn how to use your new LifeBoat account, please follow these steps.

Update August 30, 2021 - 3:00PM

The restoration process is advancing for the 4 servers where backups are available: - Beaubien is at 99.9% - Peel is at 73.62% - Atwater is at 71.43% - Rachel is at 71.29% There is no update at this time for clients on servers with damaged backups. We continue to recommend that you use the new LifeBoat account to help continue your business operations. We will make additional resources available soon to you to help with the rebuild process.

Update August 30, 2021 - 9:00AM

The restoration process is advancing for the 4 servers where backups are available: - Beaubien is at 88% - Peel is at 65% - Atwater is at 65% - Rachel is at 64% For clients on servers with damaged backups, we recommend you initiate your disaster recovery plan ASAP. We have created hosting accounts in your Client Area called "LifeBoats" to help you get up and running while we explore long-term data recovery options. The impacted servers are: - Clark - Drummond - Acadie - Bernard - Bishop

Update August 29, 2021 - 11:00PM

Accounts on servers with damaged backups
With our initial data recovery process unsuccessful, we’re now recommending that all affected clients that have experienced data loss consider using a new, temporary hosting account that’s already created in their Client Area called LifeBoat. To use this account, click into it to create your emails and upload website data just as you would on your regular hosting account. Once this is done, just link your domain to this account, update your domain’s DNS to: ark1.whc.ca ark2.whc.ca Once the DNS changes propagate (can take several hours) your new account can be used to start accepting and sending new email and upload new website copy quickly. These accounts will remain free of charge until at least January 1, 2022 and are intended as a stopgap measure until a more permanent solution is found.
Accounts on servers being restored now
The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress: - Beaubien is at 62% - Peel is at 45% - Atwater is at 45% - Rachel is at 31%

Update August 29, 2021 - 5:00PM

As our team continues to work to restore accounts, please note that live updates are paused until tomorrow morning at 10AM so we can again focus on the restoration efforts. We’ll then be able to give an account of what has been accomplished overnight.
Accounts on servers being restored now
The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress: - Beaubien is at 55% - Peel is at 35% - Atwater is at 30% - Rachel is at 10%
Accounts on servers with damaged backups
The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult: - Clark - Drummond - Acadie - Bernard - Bishop Our initial attempt to repair the data on our backup servers has failed and at this point the likelihood of successfully restoring account data from these servers is very low. As a result, we are working with data recovery experts to attempt to restore data from the source server, but this process is extremely long and tedious, potentially requiring file-by-file analysis for systems with millions of files and may require months to complete. Here is how you can move forward if your account was on one of those servers:
  • If you have a local backup
    If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
  • If you don’t have a local backup: introducing “Lifeboat” accounts
    To facilitate continued business operations for our clients during this major outage and until the situation is resolved, we will be activating temporary hosting accounts for all clients hosted on Clark, Drummond, Acadie, Bernard and Bishop servers. You will be able to log in to this new account in order to recreate your emails and upload website content, without hindering ongoing recovery efforts. A simple nameserver change will be required and step-by-step instructions will be provided. These accounts are expected to be available before Monday morning.
For further assistance
Please use our Live Chat or tickets for help requests. Phone lines are temporarily closed to better support restoration efforts. We thank you for your continued patience and understanding.

Update August 29, 2021 - 11:00AM

Thank you everyone for your patience and support during what has been a very trying 24 hours. Our team has been working tirelessly to restore service to affected clients and have had some success but the news isn’t all good, so here is where we stand.
Servers that have been fully or partially recovered
We have successfully recovered the following servers: - Decarie - Rev2 - Rev (some data loss has been reported here)
Accounts on servers with high likelihood of data recovery
We have 4 systems whose external backup servers have not been impacted, and these are actively being used to stream account restorations right now. We expect to be able to fully restore all backed up accounts by Monday evening and hopefully sooner. Here is where they are in the restore process: - Beaubien (50%) - Peel (30%) - Atwater (20%) - Rachel (1%, restore has just been started) A full restore process for a system of this size generally takes 24 hours. As soon as an account is restored, its website and email should become available so for most users on these machines you should have your services restored before the end of the day Sunday.
Accounts on servers with low likelihood of data recovery
The following 5 systems have had their external backup servers also partially destroyed, and we have been as of yet unsuccessful at recovering their data: - Clark - Drummond - Acadie - Bernard - Bishop For clients on these machines, we are pursuing 3 recovery strategies in parallel:
  1. Re-upload your content from your own backup
    If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
  2. Partial data recovery on production servers
    We have had some success in recovering hosting accounts data files (excluding databases) for 2 of the affected servers (Acadie & Bernard for now) and may be able to perform a partial restoration of these accounts within the next 48-72 hours. This process is slow and tedious and unlikely to complete before Tuesday.
  3. Manual data recovery
    We’ve contracted an external firm to assist us with data recovery efforts on our backup servers and they started their work on Saturday night. Manual data recovery is a tedious and slow process, but we expect to get a report on what can be salvaged before day’s end. This approach will likely take the longest to result in recovered data, with a timeline as long as 1-2 weeks.
We will be posting more information along with a post-mortem once the immediate operational issues have been addressed. We have mobilized our full team to assist in this effort and we currently have dozens of system administrators, support technicians and data recovery engineers assisting with the recovery effort since yesterday. We understand this incident may seriously impact your business and are doing everything in our power to mitigate the damage. More information will follow as we have it.

Update August 29, 2021 - 10:00AM

Thank you once again for your patience. We continued to work overnight on restoring accounts. Here are the updates for each of our restored servers so far: - Rev2 server has been fully recovered and should be functioning normally - Decarie server has been fully recovered and should be functioning normally - Rev server was recovered, however, some issues remain Here are the updates for servers currently in the restoration process: - Rachel's restoration process should begin within the next hour - Peel restoration is approaching 30% completion. - Beaubien restoration is approaching 50% completion. - Atwater restoration is approaching 20% completion. Services on the above servers should be fully functional within the next 12-24 hours. Please note: We will be posting another update at 11AM for the more severely impacted servers.

Update August 28, 2021 - 9:00PM

As our team continues to work overnight to restore accounts, please note that live updates will be paused until tomorrow morning at 10AM so we can focus on the restoration efforts. We’ll then be able to give a more detailed account of what has been accomplished. If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account. While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority. Servers with more severe data loss are shown below: - Clark - Drummond - Acadie - Bernard - Bishop

Update August 28, 2021 - 7:50PM

We have now proceeded to restore the servers: - Rev - Decarie - Rev2 The following servers are being restored from disaster recovery backups and accounts are gradually coming back up as they restore (full process will exceed 24 hours): - Beaubien - Peel - Rachel - Atwater The following servers have suffered from more severe data loss and we are investigating advanced data recovery steps: - Clark - Drummond - Acadie - Bernard - Bishop If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account. While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority.

Update August 28, 2021 - 5:10PM

We have now proceeded to restore the servers: - Decarie - Rev2 The following servers are being restored from disaster recovery backups: - Beaubien - Peel - Rachel - Atwater We are still investigating the situation for the following servers and do not yet have an update at this time: - Clark - Drummond - Rev - Acadie - Bernard - Bishop

Update August 28, 2021 - 3PM

We have begun the process for disaster recovery from external backups for the following servers: - Beaubien - Peel - Rachel - Atwater Considering this is a full system backup restore, restoration can last up to 24 hours before all the data is available again. We will provide updates as they become available. We are still investigating the situation for the following servers and do not yet have an update at this time: - Clark - Decarie - Drummond - Rev - Acadie - Bernard - Bishop

Update August 28, 2021 - 1:40PM

We were able to successfully restore service to Rev2 server, which is back up and running. We are advancing in our efforts to restore other systems.

Update August 28, 2021 - 1:25PM

Systems Beaubien & Peel have confirmed working backups and we are proceeding with full backup restore. The following systems are still down: - Clark - Drummond - Rev - Rev2 - Acadie - Beaubien - Bernard - Bishop - Atwater - Decarie - Rachel We are continuing to investigate recovery strategy for these systems and will update soon.

Update August 28, 2021 - 1PM

We can confirm that multiple systems are impacted by a major incident right now. We are attempting to recover source data and restore service functionality and will post updates as soon as they become available.
We sincerely apologize for any inconvenience caused and appreciate your patience as we continue to actively work on understanding and resolving this incident. Please note: Contacting support at this time will not result in any further updates or resolutions, and may add to delays. As details become available, all updates will be communicated LIVE here.

Statement – Where are we now (September 2)

WHC News & Events
UPDATE: As of September 10, the situation has improved and we're happy to report that we have successfully recovered all available courtesy backups, with most remaining impacted accounts already restored. Consult our post-incident guide or contact support if you continue to experience issues.
This has been a challenging week. As a small business ourselves, we deeply empathize with our clients during this stressful time and our team is working diligently with our clients to restore their websites and emails. We are and will continue to be there for our clients. Our investigation to date has determined that on the morning of August 28 an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals. Without authorization the individual initiated actions that resulted in data loss on part of our backup and production servers. Within hours our incident response team had identified the issue and disabled access to the source account preventing any further damage and secured the environment. Our response plan immediately kicked into action, but data was already impacted. We have assembled a taskforce team, including internal and external experts, that has been working around the clock to recover the maximum amount of data and to assist clients’ business operations. Time is of the essence, and we are keenly aware of that. While our focus has been on addressing the data disruption, we are committed to frequent communication with our clients. To date, our team’s actions have successfully recovered over 60% of impacted client accounts. Additionally, many clients rapidly restored service using their existing local backups in conjunction with our support team, reinforcing the importance of local backups. Our taskforce is actively working with data recovery experts to help system-level storage recovery with remaining accounts. We have identified the individual and confirmed this was not a ransomware attack and there is no indication that data of any kind was ever downloaded, exported, shared, or exposed. The authorities have been notified and an investigation is ongoing. We will continue working hard and will provide timely updates. Sincerely, Emil Falcon CEO at Web Hosting Canada
You can follow LIVE updates on the situation here.

Major Incident: What happened?

WHC News & Events
UPDATE: As of September 10, the situation has improved and we're happy to report that we have successfully recovered all available courtesy backups, with most remaining impacted accounts already restored. Consult our post-incident guide or contact support if you continue to experience issues.
It’s been a tough weekend here at WHC and by this, I include our clients. I want to start by thanking all the team for coming together and working through the problem constructively and with tremendous heart and energy. Here’s the situation. Based on our investigation to date, the morning of August 28 at approximately 6AM, an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers. Within only hours our incident response team had identified the issue and disabled access to the source account, preventing any further damage. The environment was secured, the individual fully locked out, and our disaster recovery plan immediately kicked into action but damage was already done. The tally of the incident, however, was and still is important: a few major systems, including some production servers and some backup servers were damaged, with a large number of web hosting and reseller hosting accounts affected, resulting in possible permanent data loss. After nearly 2 days of tedious work and a combination of external datacenter backup restores and system-level storage rebuilding, our team was able to successfully recover (or is in the process of recovering) over 50% of those lost accounts. We can confirm that Cloud, Dedicated, Weebly and Managed WordPress accounts were largely unaffected. Unfortunately, at the moment, I can tell you that several production servers and their respective backup servers are still in an unrecoverable state and the data recovery experts assisting us believe that the recovery potential is very low. As such, the focus for these accounts has shifted from data recovery to starting fresh on new, clean accounts. In parallel we will continue to attempt to recover any data we can. For clients impacted by this incident and for which we are unable to recover backups:
  • If you have an adequate local backup: contact our support team and we will get you up and running on a new account ASAP
  • If you do not have a local backup: You will need to start over from a bare, empty account. To this end, we have activated new, free hosting accounts for each impacted domain, called LifeBoat. They are available in your Client Area now and are intended to serve as a quick, free and immediate way for you to start over. These LifeBoat accounts will remain free of charge until at least January 1, 2022.
On behalf of WHC, I would like to extend our sincere apologies to those affected by this unfortunate situation. With the cooperation of those particular clients affected by the incident, we believe that we can greatly minimise the consequences stemming from this involuntary incident. We remain committed as ever to providing you with quality and affordable hosting solutions. We understand and regret the impact that this incident may have on your business and operations. We are also grateful and moved by the outpouring of support we have received as we continue working to tackle this issue. Sincerely, Emil Falcon CEO at Web Hosting Canada
You can follow LIVE updates on the situation here.

Domain Promos & Upcoming Price Changes (August 2021)

Domains

Days are getting shorter. The madness of school supply shopping is in full effect, and parents across Canada will soon watch their kiddos return to school as their daily lives return (somewhat) to normal. It’s with the same kind of enthusiasm that we’re announcing some domain promos for Fall, starting this August! So, yay!

Let's go over the latest discounts and price changes affecting popular domain extensions...

Domain promos starting August 2021

Need another domain extension for your current website, or have a new project or business idea in mind? These domain extensions may be the perfect fit for you to increase your online presence. The following promotions apply to the first year of new domain name purchases.

Domain Extension Regular 1st Year Price Promo Price
.XYZ $19.99 $1.99
.PRO $33.75 $6.99
.BLOG $44.99 $9.99
.SHOP $54.99 $11.99
.ME $28.29 $11.99
.SITE $44.99 $11.99
.CO $39.99 $14.99
.WEDDING $55.99 $19.99
*Prices shown in Canadian dollars

Price Changes

The registry (named Verisign) for the .com domain extension will be increasing its prices effective September 1, 2021. Note: That same registry (owner of the .com TLD) has the right to increase prices by 7% in upcoming years so we highly recommend renewing for 10 years on domains you want to hold onto for the long term!

Domain Extension Old Register Old Renewal New Register New Renewal
.COM $11.99 $16.99 $11.99 $17.99
*Prices shown in Canadian dollars

Bulk Domain Pricing

Do you manage more than 25 domains in your portfolio? Reach out to one of our Account Managers and we’ll provide a pricing proposal customized for your needs as well as personalized domain concierge service.

Happy Domaining!