Casual Friday

With the amount of log analysis I had been doing at the UN, mainly for tuning firewall rules, I started noticing a series of strange patterns on the network. Many were just unusual or unexpected, but some showed clear signs of attacks. That reminded me of the incident response training I had taken, and I began a deeper investigation to understand what was going on.

One of the first measures was to set up a process for creating disk images of machines showing any suspicious behavior. I got authorization to buy an external device that let us connect several hard drives simultaneously. Since it was external, it made it much easier to plug and unplug the disks without having to open up the machines in the small lab I was building in my own office. With that, I started generating disk images and scanning for malware. I even considered using an old project I had created at OWASP back in the Brasília days, called FHR, but the FHR data was already outdated and didn’t help much. I ended up dropping that idea and started using more traditional tools, like antivirus software.

In addition to finding infected machines, the analyses started to reveal suspicious behavior on servers too. On one of them, we found several ASP pages with strange names being accessed by external users. I downloaded the code and, after analyzing it, concluded someone had managed to insert those pages onto the server as a way to maintain persistent access. They allowed various actions: uploading, downloading, running commands, listing directories, among others.

In another case, while analyzing DNS logs, I identified a machine that kept trying to reach a domain that resolved to the loopback address (127.0.0.1), used by a system to refer to itself. That didn’t make much sense, so I started investigating. We found the script responsible and managed to clean up the machine. What intrigued me was why malicious code would try to connect to a local address.

The next day, once I was home from work, I decided to test the domain in question again. To my surprise, it resolved to a valid IP. That’s how I understood the mechanism: the DNS record was actively managed by the attackers. When they wanted to access the compromised server, they’d point it to a valid external address. The rest of the time, the domain was configured to resolve to loopback, making the communication seem harmless. The most curious thing was that these changes happened outside the UN headquarters’ business hours, indicating a pretty targeted attack.

After we collected enough information, the CISO decided it would be important to report the case to the FBI. He asked me to compile the data, and I sent a set of logs that could be shared. A few days later, he called me in for a meeting at the main building to discuss the matter. Beforehand, he told me it would be best to wear a suit, to avoid standing out from the rest of the participants. The meeting itself wasn’t very productive. The FBI was more interested in collecting information than in sharing anything that could help us with the investigation or with mitigating the problem.

Speaking of formal attire, when I arrived in New York I asked what the dress code was at the UN. I knew many people wore suits daily, but I was told that in the IT area the standard was “business casual.” I looked into it and initially interpreted it to mean wearing at least a tie, with no need for a suit. Over time I realized that, in practice, a shirt alone was enough. I was even the target of some jokes on a Friday for being too formal, while the rest of the folks adopted the “casual Friday” style. I ended up adapting to it too and started working in jeans and a t-shirt on Fridays.

It was on one of those Fridays, right after the end of the UN General Assembly (an extremely intense period for everyone), that I got an email inviting me to a thank-you ceremony led by the Secretary-General. Since I had never heard of such an event, I asked a few colleagues and no one knew for sure what it was about. I decided to accept the invitation anyway, figuring that, if it had been sent by mistake, someone would let me know.

At the end of the day, I went to the headquarters building, as the ceremony would be on the 36th floor, where the Secretary-General’s office is. I thought it was strange, but I kept going. It was a Friday and I was wearing jeans and a green-and-white striped t-shirt. When I arrived, a security guard informed me I couldn’t enter with a backpack, but he was kind and pointed me to a locker where I could leave it. With that resolved, I went to the ceremony location. Right at the entrance, Secretary-General Ban Ki-moon and his wife were greeting the guests. I looked around and everyone was in suits and ties. I clearly stood out. There was nothing I could do at that point. I got in line, greeted the two of them, and moved on into a room where there was a cocktail reception.

I spent some time trying to find a more discreet corner, avoiding drawing attention. The only person who also looked a bit out of place was someone wearing what looked like North Korean attire. After a while, I decided to leave. To this day, I think that invitation wasn’t meant for me. It didn’t make sense for me to be the only person from my team there, while neither my boss nor the director had been invited.