We’re Up Kinda
Posted by Chris Leckness on 06/2/08 in Site Items
Saturday, the Planet had an electrical fire that caused an explosion knocking out power to their data center. Since the Planet is the largest hosting company in the US, a lot was affected by this. Somewhere in the neighborhood of 9,000 servers and ours being one of them. We were down all night Saturday and all day Sunday and just came back online sometime this morning. I am glad to be back and I apologize for any inconvience this may have caused any of you.
We are now back up, but the site seems to be running a tad slow now and I am in the process of trying to figure that out.
Here is the most recent from the Planet. Pretty scary huh?
There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.
We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we’re expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.
Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.
We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.
We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.
I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.
I plan to have an audio update tomorrow evening.
Until then,
Douglas J. Erwin
Chairman & Chief Executive Officer
Written by Chris Leckness · Filed Under Site Items
Tagged:
Did you like this post?
Share your vote!
(No Ratings Yet)
Leave a Reply