Printed from http://www.electronista.com

Amazon EC2 data center failure affects Netflix, Instagram

updated 09:13 pm EDT, Sat June 30, 2012

Storms in North Virginia takes out Amazon data center

Netflix, Instagram, and other web services were hit by an Amazon Web Services outage. Historic thunderstorms in Northern Virginia caused power outages for Amazon data center in the region, forcing services depending on EC2 from it to be severely affected, with Netflix in particular suffering considerable amounts of downtime.

At the time of writing, the Amazon Web Services status page shows the power issues for its Elastic Compute Cloud as Resolved after lasting for 20 hours. Amazon CloudSearch, Elastic Beanstalk, and Relational Database Services in Northern Virginia were also affected, forcing the same services offered in other data centers around the world to pick up the extra work.

Power providers to the area warn of sporadic outages in the vicinity of the server farm over the next several days, as power is gradually restored to the nearly 500,000 who lost it during the storm. Full restoration of the power grid in the area could take upwards of a week.

MacNN has spoken with an electrician that works with the service. An electrical breaker with a safety interlock fused shut during the storm. This breaker being stuck prevented the diesel generator's breaker from closing, as only one of the breakers can be shut at any given time, so no backup power was available to the AWS service.

Amazon suffered a similar outage in August last year, again affecting the Northern Virginia data center. More services were affected at that point, which caused Foursquare, Reddit and Quora to work on spreading their services across multiple locations and have a more robust backup-plan for data center failures. [via TechCrunch]



By Electronista Staff
toggle

Comments

  1. msuper69

    Professional Poster

    Joined: Jan 2000

    -8

    UPS anyone?

    They do make data center grade UPSs.

    This just shows that Amazon, et. al. aren't serious about keeping their systems up during storms.
    Bad management.

  1. The Vicar

    Junior Member

    Joined: Jul 2009

    +2

    The Cloud

    Everyone with actual experience in the IT industry has been saying since the beginning that "The Cloud" is a dumb idea with this being one of the possible ways it can go wrong. This is only one of the first times it happens. Eventually, people will get the message that the whole idea is just plain dumb.

    I just wonder whether that will happen before or after the "Cloud security failure" scenario happens in real life. That one will be... interesting.

  1. neocyberdude

    Fresh-Faced Recruit

    Joined: May 2000

    +10

    UPS/Generators

    Yes, they have VERY large UPSs with entire rooms full of batteries, but just like your home UPS, they are designed to ride through short power outages (15-20 minutes or so). After that they have diesel generators that can carry the load for another few hours, but this was a very long power outage. Theoretically they could have brought in truckloads of diesel fuel, but I imagine they were counting on the other data centers to pick up the traffic load. It didn't work, apparently.

  1. fizzy

    Fresh-Faced Recruit

    Joined: Mar 2009

    +5

    regional storm

    This was a very large, and unexpectedly violent storm. The outages were not limited to N. Va. Power and networking were disrupted in the whole Mid-Atlantic area. Amazon could have had perfect power backup and still been disrupted, as could any business running their own servers anywhere in that area.

  1. chas_m

    Moderator

    Joined: Aug 2001

    +5

    good point, fizzy, but

    really these companies should all spread their data centres around a bit more. Like, you know, Apple.

  1. Grendelmon

    Dedicated MacNNer

    Joined: Dec 2007

    +1

    You all can speculate

    ...as much as you want about how "poor" Amazon manages it's infrastructure, but unless you work there, you have no idea what happened. I have yet to work with a (large) company that does NOT take it's disaster recovery seriously. And for a company like Amazon, even to resell their data centers to other providers, has got to have a serious plan in place. The storms that occurred were historic, Let's not forget about the fact that a dozen people were killed across multiple states, which nobody apparently finds important aside the fact that "Instacrap" went down. And NetFlix. Oh damn, what will the world do for a day or two.

  1. AngryFanBoy

    Fresh-Faced Recruit

    Joined: Aug 2010

    0

    Netflix?!?

    Oh horrors! I won't be able to stream Archer for several hours :(

    What to do???

Login Here

Not a member of the MacNN forums? Register now for free.

toggle

Network Headlines

toggle

Most Popular

Sponsor

Recent Reviews

SMS Audio Sync Sport on-ear headphones

When hitting the gym or going out for a trail run, headphones can cause a number of problems. From the ear buds getting slimy with swe ...

Adesso Xtream S3B Bluetooth speaker

Finding a speaker purpose-built for a specific need is challenging. Even when a Bluetooth speaker can be paired with a mobile device, ...

JBL Synchros E40BT headphones

For all the different configurations of headphones on the market, it's always a tough choice for buyers to get something that is just ...

Sponsor

toggle

Most Commented

 
toggle

Popular News