Hurricane Sandy - The Evacuation

In October 2012, my mother decided to come and visit us in New York. Since she was going to spend about a month there, she preferred to stay in an Airbnb-type apartment on her own. The apartment was close to where we lived, about a ten-minute walk away. By late October, it was already almost time for her to go home, and reports started appearing on TV about a hurricane that was causing damage in the Caribbean and had a good chance of hitting the U.S. East Coast. It’s rare for a hurricane to reach New York, but it happens.

In the last weeks of October, news started coming in that Hurricane Sandy had a strong chance of passing through New York, and a lot of preparation got underway for its arrival.

The UN is always very concerned about the physical safety of its staff and began telling us how to prepare for the hurricane. One of the things they suggested was to have flashlights, batteries, a battery-powered radio, and water on hand. We bought some of these things and left them in a drawer. At work, there wasn’t much preparation for this kind of event yet, nor were there adequate communication systems for remote coordination. The UN relied heavily on the phone system, which was based on VoIP but limited to the offices, and on BlackBerrys as the main means of communication. A more cautious colleague organized a Webex meeting so we could talk if the need arose. That was all the preparation we had for the hurricane’s arrival.

During the weekend, Gisele went out shopping and I stayed home with the kids. I was following the news on TV when I saw the Mayor of New York City announcing an evacuation order that included areas in Manhattan, where we lived. The city had areas defined in three categories with respect to flood risk. The highest-risk areas were to be evacuated. Since we lived right by the river, our building was classified as being in a high-risk area and we were included in the evacuation order. It was also announced that the subway and buses would stop running and that most tunnels and crossings to and from Manhattan would be closed or restricted.

Since Gisele wasn’t home yet, I started getting things ready with the kids. We packed bags, grabbed some things we thought we would need over the next few days, as well as some documents. I started searching for hotels outside the areas being evacuated. I managed to find a hotel in the middle of the island, the Hotel Pennsylvania, which was outside the evacuation area and had rooms available, including family rooms, where we could all stay together, which came out a bit cheaper. I booked three nights, which was the expected time for the hurricane to affect the city.

When Gisele got home, we were ready. All we had to do was grab our things and catch a bus to the hotel. The hotel was huge and quite busy, right across from Penn Station, an area normally busy with people and cars all day long. We got there in the mid-afternoon and stayed in the room until early evening, when we decided to go out to grab something to eat. Between our arrival and going out for dinner, the situation had changed a lot. There were already fewer people on the streets, the streets were almost empty of cars, and the wind was strong enough to almost drag my daughter, who was 11 at the time. It was the beginning of the effects of the hurricane. Many shops and restaurants were already closed, which was unusual for that area. We managed to grab some fast food and went back to the hotel, since the recommendation was to stay inside. In the room, we could hear the wind hitting the window and see the rain, but that was it.

In the middle of the night, my phone rang: it was some people from work. The hurricane was causing flooding all over the island and, since the UN Headquarters building sits on the edge of the river, water was already starting to come into the lower floors. The information at that moment was that water had already reached the floor just below the Data Center and was still rising. There was a Network Operations Center (NOC) team that worked 24/7 shifts inside the Data Center. With the whole city shut down, those people were stuck there, with no way out.

As soon as they let me know the situation, I found myself a corner in the room while everyone else was sleeping and connected the laptop to the internet via my cell phone. I joined the Webex session I mentioned above and a good part of the team was already there. Not everyone managed to join, because many people had no power or no internet. The situation was chaotic. The water kept rising, and the decision was made to shut down the whole Data Center and migrate whatever was possible to the secondary Data Center. The secondary Data Center had limited capacity, so we could not migrate everything. We had to choose.

Despite the chaos, the decision was made to migrate the financial management systems, which at that moment we considered to be the most important. After the hurricane passed, it became clear that this decision hadn’t been the most appropriate one, but at that moment we focused all the resources on that task. As time went on, more news came in that the water was still rising. At one point, they said it was only 50 centimeters from reaching the floor of the Data Center. We then started shutting down everything possible, even without finishing the migration. If water got into the Data Center, it would be a disaster, on top of the risk to the people who were there.

The machine shutdowns happened in two stages. First, we began shutting down the systems. Then we told the people who were physically in the Data Center to go to each machine and unplug the power cable, to avoid any problems in case of flooding. It was slow work that required enormous coordination. All communication happened through the Webex chat created by a colleague, which ended up saving the day. It was a primitive chat with only basic features, but it was what we had available.

It was almost morning when someone announced that the water had stopped rising and hadn’t reached the Data Center. It was a relief, because we still hadn’t managed to copy all the data we needed before we could shut down the storage system. The copy was taking much longer than expected.

Day broke and, with the situation a bit calmer at work, several of us were able to take a break. In my case, my family was waking up and we needed to figure out a way to have breakfast and check how the city was doing. We managed to have breakfast and went back to the room, because there was still plenty of wind and a bit of rain. We tried to get in touch with my mother, who was still at the apartment she had rented, but she wasn’t answering. We turned on the TV to see the news and try to understand the state of the city. That’s when we found out that the southern half of Manhattan had no power. From the information we got, the building where my mother was staying was in the area without power. That explained why she wasn’t answering: if there was no power, there was probably no internet either.

I then decided to go to the building to see how she was doing. My son, who was about 14 at the time, came with me. Along the way, we checked which shops were open, to decide where we could eat over the next few days. We got to the front door of the building and it really was without power. We managed to get in and went up six flights of stairs to my mother’s apartment. Everything was fine, but she didn’t know where to go or how to get to our hotel. My son and I went down with her bags and took my mother to the only possible place: our hotel room. Now there were five of us in that room. It was still okay, but it started to feel a bit cramped.

At the same time, at work, it was time to turn everything back on in the main Data Center. The water had already receded and the Data Center was no longer at risk. Again, the work needed to be coordinated: the system layers had to be turned back on in the correct order to avoid problems. Another important thing was that the people who were physically in the Data Center were able to leave and be replaced. It must have been a relief for them, after spending the whole night running around the Data Center unplugging power cables from all the servers.

We had survived our first hurricane. We thought everything was settled and that now it was just a matter of waiting for the city to go back to normal. Little by little, we realized it was still going to take a while for things to get back to normal. The subways and buses that had stopped were still not running, and there was no estimate for when service would resume. Many shops still hadn’t been able to reopen. Getting to and from Manhattan was still difficult.

Note:

• Video of what the streets around our building looked like during the hurricane.
• See other photos of the blackout at New York Magazine