Theplanet's datacentre - Largest dedicated server provider down

The planet's main datacenter is down due to fire . According to their official update there was an explosion that caused due to a short circuit and that no one was injured and no data of customers were lost.



Update Note from them

"This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock. We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well. There is no impact in any of our other five data centers."



As you know, we have vendors onsite at the H1 data center. With their help, we've created a list of equipment that will be required, and we're already dealing with those manufacturers to find the gear. Since it's Saturday night, we do have a few challenges. We are prioritizing issues as follows:
Getting the network up at H1 is first and foremost. We're pulling components from our five other data centers – including Dallas – which will be an all-night effort.
Getting power back to the data center is key, though it is too early to establish success there.
Because Server Command is in H1, our legacy EV1 customers are blinded about this incident. We are in the process of moving the Server Command servers to other Houston data centers so that we're able to loop them into communications.
We absolutely intend to live up to our SLA agreements, and we will proactively credit accounts once we understand full outage times. Right now, getting customers back online is the most critical.


Labels:


5 Comments:

Anonymous scuzzmonkey said...

I wonder what the dollar value of the repairs will run? I'm sure insurance covers this kind of thing, but I'd love to see hard figures like in one of those mastercard commercials: Structural damage: $15000 Melted hardware: $70000 Halon refill: $however much halon costs Real-Life Slashdot effect: Priceless

7:05 PM  
Anonymous PHH said...

Being in the power systems engineering biz, I'd be interested in some more information on the type of building (age, original occupancy type, etc.) involved.

To date. I've seen a number of data center power problems, from fires to isolated, dual source systems that turned out not to be. It raises the question of how well the engineering was done for the original facility, or the refit of an existing one. Or whether proper maintenance was carried out.

Properly designed systems should never result in any fault to become uncontained in this manner.

7:11 PM  
Anonymous potato3 said...

REPLY FOR PHH:



I'm a mechanical/electrical engineer by training, and what you're saying makes no sense to us. Mistakes are made in the laboratory, where things are allowed to blow up and start fires. Once you hit the real world the considerations are *very different*. While it's possible that this fire could be caused by something entirely unforeseeable (unlikely given our experience in this field), it's also possible that this was due to improperly designed systems.

I don't suppose you'd be singing the same tune if this was a bridge collapse that killed hundreds. There's a reason why engineering costs a lot, and that's directly correlated to how little failure we can tolerate.

7:13 PM  
Anonymous mrcdeckard said...

The only thing that I can imagine that could've caused an explosion in a datacenter is a battery bank (the data centers I've been in didn't have any large A/C transformers inside). And even then, I thought that the NEC had some fairly strict codes about firewalls, explosion-proof vaults and the like.

I just find it curious, since it's not unthinkable that rechargeable batteries might explode.

7:14 PM  
Anonymous gazzonyx said...

"THE ONLY THING THAT I CAN IMAGINE THAT COULD'VE CAUSED AN EXPLOSION IN A DATACENTER IS A BATTERY BANK (the data centers i've been in didn't have any large a/c transformers inside)"

Actually, modern batteries should be sealed valve or Absorbed Glass Mat (AGM) that don't vent (too much) hydrogen. During a thermal runaway, they vent a tiny bit before killing themselves, but hydrogen doesn't become explosive until the concentration in an enclosed environment is ~4%. 4% of a data center is a fairly large area. I've heard of this happening in one data center where the primary and fail over (IIRC) HVAC units failed and no one had been on site for well over a month. IOW, every battery in the place started venting and it took over a month without any air circulation for it to get to 4%.

7:18 PM  

Post a Comment

Links to this post:

Create a Link

<< SEO Blog Home