The Network Is Down!

One of the projects I took on at the UN was the rollout of web proxies at the organization’s offices. The idea was simple on paper: every web request would go through those boxes, letting us control what could be accessed and by whom. The project started in a funny way. My boss called me in, showed me a pile of devices, and explained that all of it had been bought and had to be installed before the warranty ran out.

We drew up the plan, defined standard configurations, and began the testing phase. While we worked, we kept the old system running in parallel. The problem was that, without telling me, they didn’t renew the old system’s license. Suddenly we discovered we had two weeks to migrate everything to the new appliances before the old ones stopped working. It was a network with ten thousand users, and any abrupt change could cause considerable chaos, but now there was no choice.

We managed to meet the deadline and, one morning, we redirected all traffic to the new equipment. Most of it worked well, but important bottlenecks soon surfaced. The appliances were overloaded because they had been sized years earlier, and web traffic was going through a transition: the use of SSL for encrypting web traffic was growing rapidly. Cisco (the equipment manufacturer) helped us diagnose the problem, and we found an improvised but effective solution: we installed the same software on virtual servers and created a load balancing setup between physical and virtual machines. That kept the environment stable until we could buy new, more powerful equipment.

As the workload kept increasing, we managed to hire one more person, who, together with the Russian engineer we had hired to install the appliances, became responsible for installing new spam filters that had also been bought previously. Those caused us fewer headaches, which let us catch our breath for a few months.

At the same time, I was officially made responsible for the local network of the entire UN campus in New York. That included the headquarters building, the other administrative buildings in New York, the metropolitan network connecting all of this to the two data centers, the wireless network, and the long-distance links to remote offices and partner organizations. UNICC, which had been managing the networks for years, provided day-to-day operations and had quite a bit of operational freedom, though the ultimate responsibility was still mine.

It was during that period that I experienced one of the most memorable situations of my career. I was in the offices of the UN pension fund for a meeting. I was chatting with a very nice Portuguese network administrator who was telling stories from when he had lived alongside Brazilian soldiers in East Timor. In the middle of the meeting, my phone rang several times, but I let it ring because I don’t like to interrupt meetings. When I left the building and started walking toward the headquarters, I saw the accumulated messages. They were from my best friend at work, all urgent. The entire network was down.

Since I was already close to the data center, I ran over there. The team had made good progress on the investigation and was in the process of bringing the network back, but it still took a few hours to restore everything. The next day, there was a meeting with the new area director, who had been on the job for only two weeks. The goal was to understand what had happened and how to avoid a repeat. I was still carrying some guilt for not having answered the phone, even though I knew it would have made no difference in practice. After the director’s opening remarks, I raised my hand and said that, as the person responsible for the network team, the ultimate responsibility was mine.

He responded calmly that the meeting wasn’t about finding culprits, but about finding solutions. Still, that moment stayed with the director in an unexpected way. Some time later, on the day of my farewell, he told that story in his thank-you speech and said it had shaped his image of me. Not as someone seeking to shoulder blame out of vanity, but as someone committed to what they led, aware of their own role, and willing to carry the weight of the team’s failures. To him, that was professionalism in practice.

My intention was never to stand out. I just wanted to make it clear that, during my tenure, the area’s problems were my problems. And that ended up being one of the most lasting lessons of my career. Owning a mistake, facing reality, and working to fix what caused the problem are always more effective than trying to hide it.