Who Changed My Password?

A little while after I got to New York, serious problems came up with the UN’s DNS. The UN used BIND, whose configuration was based on text files that defined its zones. A typo could break everything. The risk of human error was huge, and mistakes did happen. There was also an issue with the zones’ serial numbers, which, if not incremented, made the servers ignore changes. DNS and NTP ended up becoming my responsibility, along with building redundant infrastructure for those services. I wrote scripts to automate edits, update serial numbers, replicate zones, and restart services safely. Using the scripts reduced the incidence of human error and improved the system’s stability.

One day, the director (my boss’s boss) asked me whether I knew Active Directory. I said no; I had never administered it. He looked at me with what seemed like a mix of concern and pity and told me I’d be the person responsible for the system and the team. And that’s how I ended up responsible for the UN Secretariat’s Active Directory: eight domains split across two forests, four or five data centers, fifty thousand users spread around the world, and connections with many other organizations. It was a complex, old, brittle structure.

Right at the start, I discovered a serious problem: roughly once a week, the password for a domain admin account was changed for no apparent reason. No one knew why. Everything pointed to an attacker with very high privileges inside the AD. When I became head of the area, they gave me a domain admin account. On a Monday morning, I found out my password had been changed over the weekend. I was sure I hadn’t forgotten it. On top of that, I got a call from the treasurer’s office asking why my account was trying to access one of their machines. I knew it wasn’t me. The conclusion was obvious: there was a ghost on the wire — or rather, a hacker on our network.

We started a joint investigation with Steve, the AD administrator. We pulled logs, reviewed policies, and set up additional monitoring. The first step was to isolate the domain controllers behind a firewall with extremely tight rules. We created a new network and moved the servers there. Configuring a firewall for Active Directory is a nightmare: a few fixed ports and dozens of dynamic ports used by MS-RPC. The firewall claimed to understand MS-RPC, but didn’t really understand it very well. We at least managed to limit the range of those ports, which was helpful.

The situation improved, but there were still strange events. One of the tricks used by the attacker involved replacing the executable triggered by the accessibility button on the Windows Server 2012 login screen. That button let the user access tools like the magnifier or screen reader and could be used before authentication. When clicked, the system called an executable with high privileges. If the attacker replaced the accessibility executable with a copy of cmd.exe, they’d get a command line with permission to do almost anything on the server. We removed the attacker’s changes, disabled the feature, and limited who could access the domain controllers remotely.

We then called in Microsoft Consulting Services, which confirmed the seriousness of the problem. The consultant was categorical: once an attacker reaches domain admin privileges, you can never be entirely sure they’ve left. The recommendation was to rebuild AD from scratch, create a dedicated domain just for the domain admins, and use smart cards for authentication, but we didn’t have the team or time for that. Instead, we replaced all the servers, upgraded the software versions, tightened firewall policies, and from then on, the strange access attempts stopped. We never had any idea who the attacker was, or how long they had had access to the network.

There was still another problem that stemmed from a decision made back at the start of the infrastructure: using BIND as the DNS server, even for the Windows infrastructure. Although both use compatible standards, integrating Active Directory and BIND is not simple. For a while, Steve had been dealing with serious replication problems between the domain controllers. He suspected DNS, but didn’t understand either DNS or BIND.

Other engineers tried to help, and sometimes we managed to stabilize everything, but the problem always came back. I managed to convince my boss that we needed to get a better support contract with Microsoft. The consultant they sent really understood DNS and AD. He spent several days collecting information, mapping the infrastructure, and rebuilding the links between the servers. In the end, he restored replication, stabilized the environment, and drastically reduced the risk of data loss.