BMS - What’s New?

BMS had a long-standing and very strong partnership with IBM, which often gave the company easy access to newer technologies. One of those novelties was the virtualization of Intel servers using VMware. Virtualization technologies, which allow you to simulate several machines on a single physical piece of hardware, were common in the mainframe world, but practically nonexistent on the so-called open systems. Intel servers had finally reached a level of capacity and performance that made it viable to partition a single server into several virtual machines, similar to what had been done for decades on mainframes. It was one of the big novelties of computing at that time.

The folks at IBM gave a presentation on that technology, which caught the attention of my manager, known for a certain nostalgia for the mainframe days. After that presentation, Fernando gave me the task of specifying and acquiring a server so we could try out this new approach. That’s what we did: we bought a server with good processing capacity, high-capacity external disks, and installed one of the first versions of VMware.

Next, we had to learn how the system worked and how that new environment would be managed. It was decided that this first server would be used exclusively for creating testing and development virtual machines, never for production. As soon as we mastered the basics of the tool, it was obvious how simple and quick it was to create a new test machine: in a few clicks, the instance was ready to use. Deleting a machine and starting over was equally trivial. Without realizing it, we ended up creating a mismatch between the testing and production environments. While in the virtualized environment everything was fast and flexible, in the production environment you still had to buy physical servers, install them in a rack, do the cabling, and await for everything to be configured, a much longer process.

From then on, it was clear that large-scale virtualization would be inevitable. It no longer made sense to keep the traditional model of one physical server per system. It was one of those projects that was so successful it ended up creating a new problem: the management of the virtualized environment was so efficient that nobody wanted to go back to the old method. As people used to say at the time, though, you don’t create virtual machines out of thin air. You had to invest up front in hardware and installed capacity before you could create new instances. That took time and considerable investment until the company reached a comfortable point.

I also remember a colleague, responsible for managing Cisco equipment, who identified very early a problem that is now a reality at practically every company: since it became extremely easy to create new servers, there would be an explosion in the number of virtual machines, making it harder and harder to manage them properly and even to know which ones were still being used and which had been abandoned, consuming resources for nothing.

Besides the issues related to server management, we also faced serious limitations in the capacity of the communication links between BMS and Belgo’s plants. No matter how much the capacity of the channels was increased, congestion seemed inevitable. Belgo’s main system was SAP, centralized in Belo Horizonte. Employees anywhere in Brazil needed to access that system for things like human resources, inventory control, and invoicing. When the links got congested, the system became slow or unstable, slowing down and disrupting work.

So we started looking for alternatives to improve remote access to SAP. One of the solutions presented was a Traffic Shaping product, a technology that lets you prioritize certain kinds of traffic on a communications channel. We ran a test at one of Belgo’s sales offices in Belo Horizonte, and the experience was positive. However, when discussing the costs of rolling it out across the whole company, it became clear the solution was not economically viable: the equipment was extremely expensive and would cost more than simply doubling the capacity of the entire network.

During those tests, I dove deeper into the concepts of quality of service and traffic prioritization on networks. That’s when I discovered that Linux already offered this kind of functionality directly in the kernel. It was possible to use these native OS capabilities to optimize the use of communication links. I managed to implement a solution that controlled the bandwidth allocated to each application on the network. The process was more complex than using an off-the-shelf product, but it worked.

It remained to solve the high availability problem: what would happen if that server failed? The solution we found was to use two servers in a cluster, with heartbeat and IP take-over. We tested that configuration in the lab and the results were good enough to start deploying it at several of Belgo’s offices and plants.

Little by little, these projects let me get to know the company better, but also showed me, in very concrete ways, how certain dynamics of corporate politics work. Those lessons ended up directly influencing other projects I developed during my time at BMS — but that’s a topic for another chapter.