At Leaseweb we are always looking for ways to automate tasks and processes to make the work of our engineers easier and allow them to spend more time on innovation rather than repetitive tasks. One of those tasks is the installation of (monthly) recurring security updates (Windows, Linux, etc.).
Not every company has the latest security patches installed, as evidenced over the last view months with the ransomware attacks (WannaCry for example). Fortunately, within Leaseweb we were up to date and not impacted by the ransomware attacks, but it does highlight the importance of these security updates.
The events also raise the question if some companies did not have all their updates in order because the tedious deployment task of such patches may not be on every engineer’s “favorite things to do” list.
Within our product engineering department, we also struggle with the task of deploying these patches each month due to the amount of work that goes into deploying these updates on all our systems. For this reason, one of our teams decided to automate this job as much as possible.
At first we considered using Chef which was already heavily used in our company; however, we found that it was not the best option particularly for Windows updates.
After conducting more research, we selected Ansible. For those unfamiliar with Ansible, it is a simple yet powerful open source automation platform. This product can help with configuration management, application deployment and task automation. Unlike Puppet or Chef, it doesn’t use an agent on the remote host. The main reason we chose Ansible is it already has a nice way to deploy patches on Linux boxes via SSH, but also has a module for Windows updates.
We then set out to configure Ansible for Windows and Linux updates. Linux was easy. It only required SSH connections from a specific system, which in our case was an Ansible management server, and then running a playbook. But for Windows it was a bit more challenging. For this system Ansible uses Windows remote management services or WIN_RM. To set this up you need to enable WIN_RM, open some firewall ports, and start the services.
Fortunately, Ansible provides a nice PowerShell script to do all these tasks for you. We did discover a downside when trying to remotely run this PowerShell script. It failed because it needed to run with elevated security permissions, and Windows 2016 prompts the user to confirm these elevated rights executions.
Unfortunately, not all our Windows servers have Chef running, so we had do come up with a different way of enabling WIN_RM and opening firewall ports. In the end we decided to spread the workload of running this script amongst some team members because as the saying goes, “many hands make light work.”
The end result is that we now have both Windows and Linux servers ready to receive patches from Ansible, saving our team time which can be used for other tasks instead of deploying these patches every four weeks.
We still have some work left to do such as completely removing the remaining manual actions. At the moment, someone still needs to login to an Ansible jump host (which we use to connect to different networks) and run the playbook. We also have to manually put systems in downtime in our monitoring platform.
One other important thing we still need to implement is a result check in order to know if a deployment was successful or not.
But now that we have more time on our hands, I am confident it will get done.