Kernel live patching is a popular solution for Linux servers, allowing critical kernel security patches to be applied without rebooting. This is especially useful in environments where uptime is essential.

However, the concept of live patching is often oversimplified in many guides, leaving out important details.

In this guide, I’ll take a deeper look at what kernel live patching is and how to install and use a kernel live patching service.

I’ll also highlight important considerations you should keep in mind.

I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers to get started.

Author's Note

Before diving into the details, I want to share a few thoughts.

While exploring various guides online about live patching the Linux kernel, I noticed a common theme: many of them focus on promoting the concept as a way to keep Linux servers running without reboots, often emphasizing phrases like "reboot-free kernel updates" or "reboot-free security updates".

While this is partially true, it’s not the complete picture.

Installing a kernel live patching service doesn’t mean you can ignore installing new security updates or rebooting your server, nor does it replace the need for regular security updates.

It's important to understand that live patching provides only a temporary solution by patching the running kernel.

To stay fully protected, you'll still need to update and reboot your server, and I will explain why in this guide.

What is Kernel Live Patching?

Kernel live patching is a method of applying small, targeted patches directly to the running kernel's memory without replacing the kernel binary. These patches are applied in real-time, focusing on fixing specific vulnerabilities in critical areas of the kernel.

For example, if a vulnerability is discovered in the kernel version you’re currently using, a new version with the fix will be released. Normally, you’d need to update your server and then reboot to load the new kernel. However, if you delay the reboot, your server remains vulnerable, leaving it exposed until the new kernel is loaded.

Live patching addresses this issue by patching the vulnerability directly in the running kernel, avoiding the need for an immediate update and reboot.

These patches are designed to close critical security gaps, such as those leading to privilege escalation or remote code execution. However, live patching does not include non-critical fixes, new features, or driver updates.

In simple terms, live patching is like fixing a hole in a door by covering it up without replacing the door itself. This approach keeps your server secure while avoiding the downtime caused by a reboot.

Traditional Updates

Let me start by discussing traditional updates and how you would normally update your server without kernel live patching.

When deploying a new server from a cloud provider like Hetzner, the first step after accessing the server is to run updates. This is typically done by first running the sudo apt update command, followed by either the sudo apt upgrade or sudo apt dist-upgrade command.

Using apt upgrade won’t install a new kernel version. It upgrades existing packages, but doesn’t install new packages or remove unneeded ones. This means that your current kernel will receive security patches and bug fixes, but you won’t get a newer kernel version.

If a patch is released for the current version of the kernel, running apt upgrade will handle it. But if the kernel update involves installing a newer version or additional dependencies, it won't handle it.

On the other hand, using apt dist-upgrade can install a new kernel version. It intelligently handles changes in package dependencies, including installing new packages or removing existing ones if necessary.

👉
New to Hetzner? Use my link to get free credits!

Now, let me show you how to check for security-related updates. I'll use the following command:

sudo apt -s dist-upgrade | grep "^Inst" | grep -i security

This command provides a list of available security updates.

Some updates, such as kernel-related updates, require a reboot after installation. Others, which pertain to various packages, take effect immediately without needing a reboot.

For example, on a Hetzner server running Ubuntu 24.04 version, the command returns the following kernel-related security updates:

Inst linux-modules-6.8.0-49-generic (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64])
Inst linux-image-6.8.0-49-generic (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64])
Inst linux-image-virtual [6.8.0-45.45] (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64])
Inst linux-tools-common [6.8.0-45.45] (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [all])

Next, I'll check my current kernel version using the uname -a command:

Linux ubuntu-4gb-fsn1-2 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

This indicates I'm running kernel version 6.8.0-45, while the updates are for 6.8.0-49, a newer version in the same kernel series (6.8.0). It's an incremental update, not a major version change.

Because the older version (6.8.0-45) is already installed, both sudo apt upgrade and sudo apt dist-upgrade can apply the update.

The command sudo apt upgrade typically updates packages that are already installed, replacing the older version with the newer one. In this case, it will update the kernel packages to 6.8.0-49.

However, since these updates are kernel-related, a reboot is required for the changes to take effect.

For now, I will update all available packages. After the update process was finished, I saw these lines:

Pending kernel upgrade!
Running kernel version:
  6.8.0-45-generic
Diagnostics:
  The currently running kernel version is not the expected kernel version 6.8.0-49-generic.

I need to reboot the server for the new kernel to take effect, but I won't do that right now, as I want to show you what a kernel live patching service can do.

Kernel Live Patching Service

I will be using KernelCare Simple Patch from TuxCare, a kernel live patching service that costs only $2.95 per month. It provides security patches for a range of popular Linux kernels, which can be applied without rebooting the server.

When a new vulnerability is found in the Linux kernel, TuxCare creates a live patch to fix it. The patch is first tested in TuxCare’s internal server farm and then gradually moved through various testing tiers to ensure it's been thoroughly tested.

Once the patch is ready, servers using KernelCare will receive and apply the patch live.

You have other options, such as the Ubuntu Livepatch Service, but I personally prefer KernelCare. It is more affordable and works reliably. While the Ubuntu Livepatch Service is available for personal use for free, it is significantly more expensive than KernelCare for commercial purposes.

Now, access your Ubuntu server and run the following command as root to install KernelCare:

curl -s -L https://kernelcare.com/installer | bash

Then, register your key (which you received by email after purchasing and can also find in your customer dashboard) using the command:

kcarectl --register [KEY]

KernelCare will automatically check for new patches every 4 hours. For now, I'll manually check and update with this command:

kcarectl --update

I received the following output:

Downloading updates
Patch level 4 applied. Effective kernel version 6.8.0-47.47
Kernel is safe

As you can see, a vulnerability was fixed in kernel version 6.8.0-47.47.

Remember, I haven't rebooted yet, so my kernel version is still 6.8.0-45 if I run the uname -a command. Although I updated to 6.8.0-49, I didn't reboot, so the underlying kernel remains the old one. This means the update hasn't fixed the vulnerability yet.

Now, let me run the kcarectl --patch-info command:

OS: ubuntu-noble
kernel: kernel-6.8.0-45.45
time: 2024-10-24 07:30:13

kpatch-name: ubuntu-noble/6.8.0-47.47/CVE-2024-45016-netem-fix-return-value-if-duplicate-enqueue-fails.patch
kpatch-description: netem: fix return value if duplicate enqueue fails
kpatch-kernel: 6.8.0-47.47
kpatch-cve: CVE-2024-45016
kpatch-cvss: 5.5
kpatch-cve-url: https://ubuntu.com/security/CVE-2024-45016
kpatch-patch-url: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble/commit/?id=5017a6a30cd43240c688ed996b81a5daff2a7dc9


uname: 6.8.0-47.47

KernelCare patched our kernel live against CVE-2024-45016, which was fixed in kernel version 6.8.0-47.47 for Ubuntu 22.04 servers. Since I haven't rebooted, KernelCare has protected my server live without requiring a reboot. Great!

If I had rebooted after updating all packages, the new kernel (6.8.0-49) would have been loaded, and KernelCare would not have found any CVEs because the new kernel already includes the fix for the CVE.

For more information about KernelCare, check its documentation.

Considerations

It’s true that KernelCare helped fix the vulnerability live without requiring a reboot, but the underlying kernel hasn't been updated yet. Everything is virtual, so to speak.

If I run the uname -a command, I'll still see the old kernel version in the output. However, if I run the kcarectl --uname command, I'll see kernel version 6.8.0-47.47, which has the CVE fixed. KernelCare refers to this as the "effective kernel".

After live patching, the kernel image stored on the disk (the one used during boot) and the kernel running remain unchanged, meaning we are not fully protected. And we don't benefit from any other enhancements, such as new features or non-critical fixes.

Another thing to mention is that there are some security updates that are not kernel-related but still require a reboot to be fully applied. However, you don't receive a message prompting you to reboot like the one I got after the update process was finished.

For these reasons, I recommend updating the server regularly to apply both non-kernel security updates and any new kernel updates. This ensures that the kernel image stored on the disk is also updated, so when you reboot, the new kernel is loaded.

I will reboot the server and check the running kernel, as well as what KernelCare has to say.

After the reboot, I ran the uname -a command:

Linux ubuntu-4gb-nbg1-6 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

As you can see, the server is now running the new kernel.

If I run the kcarectl --update command again, I get the following output:

Updates already downloaded
No updates are needed for this kernel
Kernel is safe

This means there is no live patching needed because we are now running the new kernel, and there are no vulnerabilities to fix.

It is wise and important to reboot from time to time.

Conclusion and Final Thoughts

The real value of live patching is that it buys you time. It allows you to decide when to reboot while still protecting your Linux server instantly and without intervention if vulnerabilities are discovered in the kernel.

It’s a powerful tool, but one that should be used as part of a broader update and maintenance strategy.

👉
For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.