When Hurricane Sandy smashed into lower Manhattan last week, customers of the data center Peer1 faced the prospect of major downtime. Just a blackout would’ve been no problem. But when the basement flooded, it took out the pumps that transport fuel from the reserve tanks to the generators on top of the building. That’s where Squarespace CEO Anthony Casalena came in.
“I wake up Tuesday, I live in Soho,” said Mr. Casalena. “There’s no reception. There’s no power, so all the cell towers are dead.” Somehow a couple of messages snuck through to his cell phone: “Anthony there’s a major problem at Peer1, the basement’s flooded, they can’t access any reserve fuel, we have 12 hours.” He hurriedly packed a bag and started walking downtown.
Squarespace was already preparing its “hundreds of thousands” customers using the platform for the possibility of downtime. But when he got to Peer1, Mr. Casalena realized it might be possible to keep the data center going. All they needed was a steady supply of fuel to the generators on the roof. So he told the team back at the office not to shut down the service just yet–”I emailed all the people and said, don’t shut down. I don’t see them turning off, and I think we’re just going to try and push it.”
There was fuel in the area. The only problem was getting it upstairs. “So we find, like, oil drums–one of our employees finds oil drums somewhere on Craigslist,” Mr. Casalena explained. “We’re just filling them up and trying to take them upstairs, and on day one it was pretty haphazard. It wasn’t like, a nice thing with everyone on the stairwell carrying things one flight–it was like half an oil drum of diesel with people pushing it upstairs taking an hour and a half.”
Employees from Squarespace, Fog Creek and Peer1 all pitched in, first with drums and then buckets, working through the night. Their efforts were chronicled moment to moment on a bare-bones status blog.
On day two, they hired a crew of day laborers and assigned one man to each flight of stairs, managing to truck hundreds of gallons to the roof. On day three, the building went into lockdown, as crews attempted to put in place proper fixes rather than quick-and-dirty workarounds. Luckily, by that point, a Peer1 engineer had finally gotten an impromptu pump working.
As we spoke, Mr. Casalena toggled back and forth between modesty and amazement they’d actually pulled it off. First he said, “I walked down and tried to help, and once it became evident something was possible, I just asked anyone if they wanted to help, and they did, and this is the result.” Moments later, he added, “This thing could have fallen apart from like ten angles. Even if it does right now, we still bought three days of uptime. That’s just crazy.”
The can-do startup mentality rarely invovles so much manual labor. And, after all, Datagram–another major data center just down the street–went down for days, taking Gawker Media, Buzzfeed and the Huffington Post with it. But to Mr. Casalena, it seemed less like a business decision and more like he had no other option.
“I am really, really proud of Squarespace’s uptime and everything we’ve accomplished,” he said. ”So, sitting there in an apartment where there’s no electricity or anything else–I mean, I would have to be, like, so lame not to walk down to the data center and just try and help. What am I going to do, sit at home in my apartment? That’s just absurd.”
“It’s okay to care about things, you know?” concluded Mr. Casalena. “Even things as silly as websites.”
We’re pretty sure Peer1, which came out of this looking like heroic to Datagram’s hapless, would agree.
This entry was posted in Uncategorized. Bookmark the permalink