Montreal Tech Watch

Last night was a test for iWeb Technologies support team. A fire broke near their iWeb-CL data center, and as a result, power supply was spotty. In such a scenario, and to prevent any sudden power outage for hosted servers, generators are started and iWeb waits for HydroQuébec to provide again regular electricity.

iWeb was especially unlucky though. All generators started as expected, but one transfer switch didn’t switch power as expected to one of the 3 generators; so a while later, a whole segment housing 3000 dedicated servers was off the grid.

It’s a web hosting company’s worst scenario. In less than a second, thousands of websites, web applications and services, databases, e-commerce websites are down. A support team must then race to restart all servers, making sure that the restart process doesn’t overload the power infrastructure, and finally, most servers are hopefully back a few minutes later… but the damage has already been done. Data has been lost, software were stopped, hard drives might have been corrupted due to the sudden power outage. Websites are not back online since most of the time, a sys-admin must login to the server and launch the web server and other services like databases.

The incident makes us think that while software has the central role in tech, hardware and networking is still a critical component.

As for iWeb and hosting at iWeb, there are many questions arising.

First, could this have been prevented? As far as I know, no data center in the world is free from accidents. If you are a sys-admin, you just have to assume that a server WILL shut down one day, due to faulty networking, power outage, disk failure etc. In the case of iWeb, I know they do weekly stress tests, all equipment, power supply and networking are redundant, server components are tested before delivered to customers. Even if your name is Google, Apple or Amazon, you can’t have a fault-free data center, and even solutions like cloud computing can’t prevent this. You can have an image backup, but there will be outages during the time required to transfer the image and provision another instance in another data center. As far as I know, the only way to get 100% risk-free hosting is to get a hosting solutions from 2 different providers, and use 1 as a disaster recovery backup. An Internet-based business must then architect their hosting infrastructure based on those probable scenarii.

Second, in the case of iWeb, you can see here lots of drama. That’s more than 500 comments in a few hours. Lots of angry customers, a few supporting iWeb, lots of impatience etc. iWeb support team was there though, which is good, with people like “Sylvain – iWeb” replying to each customer. I am wondering though why there wasn’t any activity on iWeb’s twitter or Facebook. That’s 2 medium you absolutely have to update, track and communicate. A frustrated customer on twitter will do lots of damage, and that’s probably another customer you will have to phone in the next days to get him back.

I shall leave the conclusion of the story to iWeb’s customers. In those times of crisis, customers who were looking to switch to other providers will do so in the next couple of days. Other customers who feel that iWeb had a responsive and professional support team will stay. It’s a test for iWeb, a test they didn’t need since they were launching the new iWeb SmartServers

Photo: Photo-Media.ca