Dirty COW Linux Kernel Vulnerability Fixed

dirty-cow-blogpost
Last week a very serious vulnerability in the Linux kernel, the so called Dirty COW, was reported. Our dedicated Linux kernel team immediately addressed the issues and were able to patch it in less than 24 hours on the majority of our servers. What is more, we managed to do this without server reboot and we avoided the downtime that normally results from such kernel update activities. To learn more about the vulnerability and how we addressed it read below.

What is that Dirty COW about?

The Dirty COW vulnerability allows attackers to gain root access to servers and take control over the whole system. The security hole was detected by researcher Phil Oester, who found out a race condition in the way the Linux kernel's memory subsystem handles copy-on-write (COW) breakages of private read-only memory mappings. Attackers can use this to gain write access to otherwise read-only mappings and this way take control over whole systems. For more technical information you may check the official vulnerability page and this site which is dedicated to the vulnerability.

How widespread is the issue?

The issue most probably affects hundreds of thousands, if not millions, of Linux based machines. If you are not running the latest version of the Linux kernel you should be worried. In fact, even if you are running the latest kernel you should still be worried, as at the time this post is written not all vendors have patched their respective kernels yet. If you want to try and hack your own system you can visit this Github page and use the PoCs provided on it. According to the reports, the following Linux distro versions are vulnerable (please note that this is not a complete list but rather a list of the most popular Linux distros):

  1. Red Hat Enterprise Linux 7.x
  2. Red Hat Enterprise Linux 6.x
  3. Red Hat Enterprise Linux 5.x
  4. CentOS Linux 7.x
  5. CentOS Linux 6.x
  6. CentOS Linux 5.x
  7. Debian Linux wheezy
  8. Debian Linux jessie
  9. Debian Linux stretch
  10. Debian Linux sid
  11. Ubuntu Linux precise (LTS 12.04)
  12. Ubuntu Linux trusty
  13. Ubuntu Linux xenial (LTS 16.04)
  14. Ubuntu Linux yakkety
  15. Ubuntu Linux vivid/ubuntu-core
  16. SUSE Linux Enterprise 11 and 12.
  17. Openwrt

What is the standard issue resolution?

The easiest way to protect your computers running Linux is to update your Linux distro to the latest version. Keep in mind that this action, however, requires a reboot. You can use the following commands to update your Debian/Ubuntu and RHEL systems:

Debian/Ubuntu:
$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade

RHEL:
$ sudo yum update
$ sudo reboot

CentOS:
There is still no official update of the CentOS kernel. At this time the only way to patch your CentOS servers is to follow the instructions from this link.

Once you reboot your Linux computers, ensure that they are running the new kernel by executing the following commands:

$ uname -a
$ uname -r
$ uname -mrs

What SiteGround did to resolve this issue?

For quite some time we use our own custom kernel. This means that when issues like this appear we are not dependent on the updates released by the OS developers. Instead we can get creative immediately and we can craft our own solutions. In this particular case our devops team made use of several updates provided by the upstream kernel developers and the kernel security experts from GRSecurity related with the issue. We adapted them in a timely manner and created a patch that worked for our specific kernel.

As we mentioned, upgrading the kernel means that you will have to reboot your servers. Needless to say a hosting company such as SiteGround cannot simply restart thousands of servers and cause downtime to hundreds of thousands of sites.

That is why we decided to patch the kernel without restarting our servers. We used a tool called kpatch, which allowed us to patch the running kernels without rebooting or restarting any processes. The problem with using such tools is that there is always a chance that something may go wrong and a server may crash. We have different servers configured in different ways that run different kernels. That is why we had to test a big number of scenarios before we applied the patch live on so many servers. We knew about the problem before the Dirty COW website was created and we started working on a fix the moment the kernel developers released the patch. That is why we had enough time to perform the needed checks.

All shared, cloud and dedicated SiteGround servers are already patched.

Enterprise Cloud Solutions Architect

My challenging job is closely related to all kinds of Free and Open-Source Software products (some of my favorites are WordPress, Joomla!, Magento, Varnish and Apache mod_security). As a Web security and performance freak I am always hyper focused on solving all kinds of issues and improving our services.

14 Comments

  1. Reply October 25, 2016 / 02:59 Stuart BonellSiteGround Team

    Certainly grateful for your efforts on this. We had 4 mins downtime in the middle of the night and contacted support because with Dirty Cow in the wild we were a bit more sensitive to such things. The good news was that it was patched. A small amount of downtime like this is worth it for such an issue. Pass on my thanks to your team for coming up with the fix.

    • Reply October 25, 2016 / 08:35 Hristo PandjarovSiteGround Team

      Thanks for the kind comment. I don't really think your downtime was caused by this patch tough since we managed to apply it without causing any downtime to our servers. Maybe it was a short-term connectivity issue or something like that...

  2. Reply October 25, 2016 / 12:33 Peter ReckSiteGround Team

    I can't stress this enough: SiteGround is the BEST hosting company I have *ever* been associated with as a client!!! This ranges from technical competency, what cost-effect efficiency is extended all the way to outstanding customer support. You folks at SG can be really proud for what you offer and stand for! THANK YOU.

    Peter

  3. Reply October 26, 2016 / 01:21 Billy ParkerSiteGround Team

    I was just just about to submit a ticket regarding this, I take it cloud accounts have also been patched? Magento have released a backend notification on 25 Oct 2016 17:51:10. There maybe an influx of support tickets.

    • Reply October 26, 2016 / 01:29 Hristo PandjarovSiteGround Team

      Yes, all cloud servers are now patched 🙂

  4. Reply October 26, 2016 / 06:41 Zoran FilipovićSiteGround Team

    Awesome! SiteGround is one and only: The Best! SiteGround is: The Joy of Web!

  5. Reply November 11, 2016 / 02:47 RTLSiteGround Team

    TYVM

  6. Reply November 11, 2016 / 10:14 EoinSiteGround Team

    Phenomenal and a reason to never leave. I wouldn't even know about the issue, let alone patching it!

  7. Reply November 14, 2016 / 10:32 Jaswinder KaurSiteGround Team

    I am lucky I am using Siteground.

    Thanks to Siteground team!!

  8. Reply December 7, 2016 / 01:12 hos7einSiteGround Team

    hi
    However,Do you propose to use kpache for live paching on server?
    tnx

    • Reply December 7, 2016 / 02:29 Daniel KanchevSiteGround Team

      Hi, hos7ein 🙂 Yes, the kpatch tool is very useful and if you want to apply kernel patches without restarting your servers it is a good tool to do it. You may check the official Github page for more info:

      https://github.com/dynup/kpatch

  9. Reply December 7, 2016 / 03:13 hos7einSiteGround Team

    Hi DANIEL KANCHEV
    tnx for respond

    but in github kpatch project saying :

    WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss may occur!

    now,are you think kpatch is stable and trustworthy tool for kernel live patching?

    • Reply December 7, 2016 / 03:20 Daniel KanchevSiteGround Team

      We do agree with the developers of kpatch that it should be used with caution. For example, we first verify and test our patches on our spare servers which are configured exactly the same way as our live servers. We do perform many tests before we apply the same changes on servers that actually host our clients' sites. In conclusion I would say that the tool is useful and you can use it to apply patches to production servers but first get familiar with the tool and always test your changes on spare servers and not directly on your production machines.

      • December 7, 2016 / 05:29 hos7einSiteGround Team

        tnx Daniel for respond

        +1

Reply

* (Required)