Firewalls and DNS

When I moved to New York to work at the United Nations, I arrived with a pretty specific mission: to take care of the firewalls. There were two data centers, each with its own set of equipment, and an outsourced contractor who took care of it all. It didn’t take long to discover the “trick” he used to avoid problems: he’d left the firewalls with no rules at all. The firewall was there, its little lights blinking, but it let absolutely everything through. Whenever someone complained that the firewall was getting in the way of some system, he’d say with conviction that that wasn’t the case. And he was right. If the firewall wasn’t doing anything, it couldn’t be the culprit.

So I started working to change that scenario, which was complicated because the environment had been accumulating tech debt for years. The best opportunity came when we set up two new data centers in Europe. We bought new firewalls, and I decided to do everything right from the start: a default deny policy, explicit rules for each service, and nothing going through without being declared. Since we started from scratch, it actually worked, despite many complaints from colleagues who weren’t used to it and had no idea which ports their applications used. After that, we needed to do the same with the old firewalls in New York, which sat inside a network full of legacy systems.

The network in New York was a patchwork mess, held together by years of quick fixes and workarounds. At some point in the past, they had moved the data center and tried to rebuild everything from scratch, but when the old building had to be emptied in a rush, they abandoned the idea of putting everything on the new network and simply plugged in whatever had not yet been moved, keeping it more or less as it had been before. The result was a chaotic mix of technologies and hacks. On the network team, there was an extremely detail-oriented Vietnamese engineer who had drawn a map of the full network. When he showed me the diagram, it became the treasure map: from that point on, it was the official map of the network. Then we started planning the restructuring: removing legacy parts, consolidating everything in the new environment, and rewriting firewall rules in an organized way.

We bought a system (Tufin) to analyze the logs and find out which rules were more or less used. We created completely open rules for all existing machines and set a default deny rule for everything else. That way, we established that any new machine would only go into production with its rules defined from the start. For existing services, we observed the ports they were using and defined new rules based on those observations. After a while, once we were confident we had mapped all the ports, we turned on the blanket default deny rule network by network. That change caused a lot of resistance from other teams, to the point where people told me that I was the one who should know which ports to open, since I was responsible for the firewall. I then had to explain that knowing which network ports an application uses is part of the job of running that application. On that front, we had support from management and the CISO and were able to stand up to the pressure (but it was nevertheless quite stressful).

In 2012, it was the 20th anniversary of the UN Conference on Environment and Development (Rio 92). To mark the date, the UN organized Rio+20, a new conference focused on discussing environmental issues and assessing the progress made since 1992. As part of the preparation, a team of six or seven people from IT was sent to Rio de Janeiro to take care of the event’s IT infrastructure, while the rest of us stayed in New York with the responsibility of providing remote support to the team in the field. My team, specifically, was in charge of setting up and maintaining a VPN between the UN and the infrastructure deployed at Riocentro. To do that, we got in touch with the conference’s network provider and began coordinating the technical details with the engineer in charge, who, curiously, wasn’t Brazilian but Portuguese — a reflection of a period when the Brazilian economy was strong enough to attract professionals from Portugal.

After the VPN was set up, it looked like everything was working fine, at least at first glance. Before long, though, one of the engineers in Rio, who was responsible for the Linux servers, started reporting instability, describing the problem as frequent VPN drops. When we investigated, we didn’t find any direct sign of failure in the connection itself. The behavior we observed was more subtle and, at the same time, harder to diagnose: transmissions would start at a good speed, but gradually the transfer rate would drop until it was practically zero. Initially, the VPN was identified as the main suspect, which led us to review configurations and discuss adjustments with the Portuguese engineer responsible for the network infrastructure, but despite some changes, the problem persisted. In practice, we lived with the problem throughout the whole conference, without ever being able to identify a clear root cause or a definitive solution.

The situation ended up being escalated to the Telemar control center to check whether there was any issue with the interconnection with AT&T, which was the UN’s provider in New York. In one of the most curious moments of that process, I took part in a call between a senior Telemar engineer and an AT&T engineer, acting as the interpreter, since the Brazilian representative didn’t speak English. Even with that combined effort, the conversation led to no concrete conclusion, and we moved on without a satisfactory explanation for the problem.

To this day, I suspect the origin of the problem lay in the Linux servers themselves. The progressive degradation of the transmission rate suggests something sensitive to latency, a kind of behavior that tends to appear only on long-distance networks, like the link between Rio and New York, and that would hardly show up in local testing, where latency is much lower. That suspicion was reinforced when we discovered that the engineer responsible for the servers had a habit of changing advanced parameters and frequently recompiling the Linux kernel, which can introduce side effects that are hard to predict, especially in more complex, distributed network scenarios.

In parallel with that incident, there was also a structural problem that was becoming evident: the firewalls had been neglected before I arrived. The equipment was running old, unsupported software versions, which made it impossible to call the vendor for help when more complex problems came up. That limitation further increased the difficulty of diagnosing situations like this one, where multiple layers of infrastructure could be involved. Fixing that problem became one of my priorities, and I put significant effort into updating all the systems and making sure that, at least from then on, we would always be running updated and supported versions.

Right at the start of that effort, an external consultancy said we should replace all the Check Point firewalls with Palo Alto products, which were considered among the best at that time. I pushed back and asked what the justification would be for that change. I was told that the main reason was that Palo Alto’s firewalls had built-in intrusion detection and prevention capabilities. I then argued that Check Point also had IPS capabilities on its firewalls. I also said that, since we were starting from scratch, there was no urgent need to adopt the best product on the market and that deploying an extra module on the existing equipment was much quicker and safer. But the main thrust of my argument was that we had already invested a lot of money and time in the equipment, that we had a team that was highly skilled with Check Point devices, and that we had a good relationship with the vendor, which made a big difference when solving problems. I ended my pitch by saying we could use the funds that would be spent on the swap to buy higher-capacity Check Point equipment, improving performance and increasing the network’s capacity. Both the CISO and my boss accepted my arguments, and we began working on the purchase and replacement of the equipment.

The replacement of the firewalls with larger equipment was a major project and only ended on my last weekend in New York, but that is a story for another chapter. That period was a deep dive into the UN’s IT infrastructure, into those parts that only show up when something breaks and everyone needs an immediate solution. In the end, I felt the weight of the word “production” directly: there’s no rehearsal, no pause, no second chance. The work is live (and alive)!