Hurricane Sandy - The Day After
The airports were still closed, and no one knew when they would reopen. That meant tourists who were about to leave the city were stranded with no way out. The hotel lobby started to fill up with people who were supposed to check out but couldn’t go anywhere. The hotel staff couldn’t get to work either, so there weren’t enough people to keep the place running. Little by little, we realized that chaos had just changed shape.
That night, after spending the day trying to restore everything we had shut down, I went to try to have dinner with my family. We thought the easiest thing would be to eat at the hotel restaurant. The problem was that there was a severe staff shortage, so they weren’t able to keep up with demand from all the guests who wanted dinner. It was taking almost an hour just to place an order. We realized there was no point in staying there and went out into the street to try to find something to eat. Shops and restaurants nearby also had their own issues, and most were closed. We found a very small pizza place and went in. We managed to eat, and at the same time we realized the staff inside had probably slept right there on the floor of that little shop. They had no way to go home and were helping feed the people who were also stuck in that area.
At work, the situation improved a bit. Most of the systems had come back, but since public transportation still wasn’t back to normal, we kept working from home. Several colleagues had serious problems with power outages, no water, fallen trees… My boss had no water or power and had to drive to a local Starbucks to charge his phone. Several colleagues had difficulty getting fuel for their generators, to keep their homes heated and maintain electricity.
In our case, our building also had no power. Since we lived on the 30th floor, the apartment had no power, no heat, and no water, which depended on electric pumps to reach the apartment. The elevator was out, and the only way to get up there was to climb 30 flights of stairs. The hotel reservation was about to end, and we thought it was best to leave the room, where we were all kind of cramped together. We found an Airbnb with three bedrooms in Harlem and decided to move there.
By then, the city had a few buses running again and the occasional taxi passing through the streets. We piled into a single cab and paid a small fortune for the driver to drop us off in Harlem. When we got there, Juju realized she had forgotten her phone at the hotel. That turned into another adventure: Gisele went with her to retrieve the phone. They spent the whole day on it, but they found the phone.
Gisele’s office was closed, but people were already starting to talk about going back. Mine had been operating since the night of the hurricane, but still remotely. The kids’ school was closed because it had no power, so they stayed with my mother most of the time. My mother was supposed to have flown back to Brazil on the Monday after the hurricane, but since the airport was closed that day, the flight was canceled and she had to stay in New York, waiting for the airline to arrange another flight.
The airline was able to get her onto a flight the following Monday, a week later. At that point, we decided to leave the Harlem apartment, which wasn’t great, and head back closer to home. We found a reasonably priced apartment in our own neighborhood. The problem now was how to make sure my mother would get to the airport. The subway wasn’t fully functional, and taxis and rental cars were affected by fuel shortages. The hurricane had caused difficulties in fuel distribution across New York City, so even renting a car was risky, because there might be no gas. We did a lot of research and eventually found a limo company that guaranteed it would have a car on Monday. We paid more, but it worked.
On Monday morning, we all headed south. We had lunch, then my mother got into the limo and we continued on to our new hotel. We got word that she had boarded and that everything was fine. It was a relief. At that point, we had already been away from home for seven days. I took the kids and we climbed the 30 floors to our apartment to grab some clothes and other things we needed. When we got to our building, we saw something we didn’t expect: some elderly residents had not evacuated their apartments and now, with the elevators out, were stuck at home, with no power and no water. So there was a group of younger people putting water bottles in backpacks and climbing the stairs, sometimes 20 or 30 floors, to bring water to them.
During this period, Gisele had already gone back to the office and I also needed to start going in. But the kids’ school still had no power, so they stayed at the apartment hotel. It took another three days or so for power to be restored to our building and the school. We were able to go home, and life was more or less back to normal.
At work, some very old systems failed catastrophically when they were shut down; the other systems were already running normally. We then entered a phase of trying to understand what had worked and what hadn’t during the emergency, and what we needed to do to make sure we could survive another event of that magnitude. The first lesson we learned was that the list of systems we considered critical was completely wrong. The most critical systems in this kind of situation were the communication systems: phones, BlackBerry, and email. In an event of this nature, what mattered was being able to talk to people and coordinate actions.
We were told that systems we had considered critical, like the financial system, could be down for up to two or three weeks. In other words, we learned that it was important to talk to the business before a disaster happens. Today that seems obvious, but at the time it was something no one on the team had thought about. Another thing we learned was that we needed to improve data synchronization between the two Data Centers. That was the process that had taken the longest during the migration attempt. And the director used the opportunity to hire someone to write a disaster recovery plan.
From the IT teams’ perspective, especially the infrastructure team, we realized we needed better communication and coordination mechanisms and, perhaps, some help making sure people were better prepared at home, since many colleagues had trouble keeping their families safe during the hurricane and in the weeks after. For many people, the weeks after the hurricane were worse than the night the hurricane itself arrived.
That was probably the biggest adventure we went through, both personally and professionally, during our time in New York.