Why Linux Servers Need Reboots and How to Avoid Downtime

Rebooting a Linux server might seem straightforward, but in a production environment, it requires careful planning to avoid downtime or disruptions.

Technologies like live patching can delay the need for a reboot and minimize downtime, but reboots are unavoidable at times.

While I can’t promise you’ll never need to reboot again after reading this, I will help you do so with confidence and minimal impact when necessary.

This guide explains why reboots are sometimes required, how to handle them correctly, and how to avoid them whenever possible.

I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers to get started.

Why Does Linux Ask for a Reboot?

While Linux is known for its stability and can run for extended periods without any issues, there are specific scenarios where a reboot becomes necessary.

Here are the most common reasons:

1. New Kernel
   1. Running updates often involves installing a new kernel version, which requires a reboot to load. This is one of the most common reasons for needing a reboot.
2. Critical Updates
   1. Some security updates are not kernel-related but still require a reboot to be fully applied.
   2. Essential components, such as the glibc library or critical drivers, often require a reboot to take full effect.
3. Filesystem Changes
   1. Significant changes to the filesystem, such as modifying the /etc/fstab file, sometimes necessitate a reboot to ensure the server applies the new configurations correctly.
4. Hardware Changes
   1. Adding or replacing hardware components, such as CPUs, GPUs, or memory, usually requires a reboot to take effect.
5. Configuration Changes
   1. Modifications to key system settings, such as bootloader configurations (GRUB) or kernel parameters, often require a reboot to apply.
6. Unexpected Crashes or Freezes
   1. Issues like kernel panics, hardware faults, or other unexpected failures may make a reboot necessary to restore functionality.

These are some of the most common reasons, but you may face other situations as well that require a reboot while managing Linux servers.

Checking If a Reboot Is Necessary

Determining whether the server requires a reboot or not is your first step.

One simple way to check is by looking for the presence of the /var/run/reboot-required file using the following command:

sudo cat /var/run/reboot-required

This file is a signal used on Debian-based distributions, like Ubuntu, to indicate that a reboot is needed after certain package updates. If the file does not exist, it typically means no reboot is currently required.

Another method is to use the needrestart command:

ivan@vm1:~$ sudo needrestart
[sudo] password for ivan:
Scanning processes...
Scanning linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
ivan@vm1:~$

This command examines running services, libraries, and kernel modules to determine if any of them need to be restarted or if a reboot is required.

If your server needs a reboot, you may also see a message every time you SSH into the server, notifying you that a reboot is required.

Avoiding Downtime

In the following sections, I’ll show you some ways to avoid downtime and accomplish tasks without a reboot.

These are just examples of how I would handle these situations, but there may be cases where a reboot is unavoidable or where these methods might not work as expected.

Skipping Kernel Reboots

As I mentioned earlier, some updates involve installing a new kernel version, which often includes important fixes for vulnerabilities in the older version.

To load the new kernel and eliminate the security risks, a reboot is required. Without it, your server would continue using the older, vulnerable kernel.

However, you can use a live patching service, which allows you to patch the running kernel in real-time without needing a reboot. While the kernel image stored on the disk (the one used during boot) remains unchanged, the running kernel is patched, closing the vulnerability.

This can buy you time and help you avoid an immediate reboot, but eventually, you’ll still need to reboot to update the kernel image on the disk.

It’s also important to note that new kernel releases often come with new features and enhancements, which you won’t benefit from until a reboot is performed.

Managing Critical Updates

Some security updates are not related to the kernel, and updates to essential components, such as the glibc library, may still require a reboot to be fully applied.

In some cases, a reboot is needed because certain services must be restarted to reflect the updates. However, instead of rebooting, you can identify and restart only the affected services manually.

You can use the needrestart command, which I covered earlier, to identify which services need a restart after an update.

Once identified, you can restart these services individually, avoiding the need for a full reboot.

Filesystem Changes

If you make any filesystem changes, you could try remounting instead of rebooting.

Remounting is the process of making a filesystem or partition accessible again after it has been modified (for example, after changes to its mount options). This can be done without requiring a reboot.

Let me demonstrate that by changing a mount option for the root / partition.

First, let me check which mount options are specified for the root partition in the /etc/fstab file:

ivan@vm1:~$ sudo cat /etc/fstab 
[sudo] password for ivan:
...
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=93050c6c-d74b-9257-75fb-27312cd730f2  /     ext4  errors=remount-ro  0  1
...
ivan@vm1:~$

As you can see, we have the errors=remount-ro mount option specified for the root partition.

Now, I will add noatime to it, which will prevent the access time of files from being updated on every read, like this:

UUID=93050c6c-d74b-9257-75fb-27312cd730f2  /     ext4  errors=remount-ro,noatime  0  1

Before remounting, let's run the mount | grep ' / ' command to see the current mount options for the root partition:

/dev/sda on / type ext4 (rw,relatime,errors=remount-ro)

As you can see, currently, we have the relatime option enabled.

Next, let’s try remounting the root partition to apply the changes without rebooting:

ivan@vm1:~$ sudo systemctl daemon-reload
ivan@vm1:~$ sudo mount -o remount /

After remounting, let's check the mount options again:

ivan@vm1:~$ sudo mount | grep ' / '
[sudo] password for ivan:
/dev/sda on / type ext4 (rw,noatime,errors=remount-ro)
ivan@vm1:~$

As you can see, the noatime option has now been applied, and the relatime option has been replaced.

Tweaking Kernel Parameters

Sometimes, we need to tweak kernel parameters, such as when hardening the Linux kernel.

This typically involves adding the desired parameters with their modified values to the /etc/sysctl.conf file and rebooting the server for the changes to take effect.

However, instead of rebooting, you can apply the changes in real-time by running:

sudo sysctl -p

This command reloads the parameters from the sysctl.conf file without requiring a reboot.

Alternatively, you can use the following command to modify a parameter temporarily:

sudo sysctl -w [parameter]=[value]

Keep in mind that changes made using sysctl -w will only persist for the current session and will be reset upon reboot.

That’s why it’s best to add your parameters to the sysctl.conf file, apply the changes in real-time, and ensure they remain in effect even after the server reboots.

Reboot After Kernel Panic

Kernel panics can result in your server becoming unresponsive and require a reboot to recover.

To prevent downtime due to a kernel panic, you can configure the server to automatically reboot after a panic occurs. This can be done by adjusting the kernel.panic kernel parameter.

Open the /etc/sysctl.conf file and add the following line at the end of the file:

kernel.panic = 10

This parameter ensures that the server will automatically reboot 10 seconds after a kernel panic occurs, minimizing downtime.

After making this change, you can apply it by running:

sudo sysctl -p

This way, if a kernel panic happens, the server will not stay down for too long and will automatically reboot, ensuring higher availability.

Different Ways To Reboot

In the following sections, I’ll walk you through several methods to reboot a Linux server, depending on the situation and the level of control you need.

These methods range from graceful reboots to more forceful options for when the server becomes unresponsive.

It’s important to choose the right method based on your server’s state and the tasks at hand.

The reboot Command

The reboot command is one of the simplest and most common ways to restart a Linux server.

Just type the command as is:

sudo reboot

It performs a graceful reboot by sending signals to terminate all processes and safely restarting the server.

Use the reboot command for routine restarts in a production environment or when you need a safe and straightforward reboot option.

Reboot Using shutdown

The shutdown command can also be used to restart a Linux server. It allows you to schedule a reboot and notify users about the upcoming restart.

Use this command for an immediate reboot:

sudo shutdown -r now

The -r option is used to indicate a reboot.

You can also schedule a reboot with a delay, like this:

sudo shutdown -r +5 "Server will reboot in 5 minutes"

It performs a graceful shutdown by notifying users and stopping services before restarting the server.

Use the shutdown command for scheduled reboots or when you want to give users a heads-up before the server restarts.

Reboot Using init or systemctl

You can reboot a Linux server using either init or systemctl, depending on the system's initialization method.

The init command is used to switch between runlevels, and runlevel 6 triggers a server reboot by reinitializing all processes. It’s mainly used on older systems with SysVinit, and while it can be useful for advanced users, it should be used cautiously in production environments.

The command for a reboot using init is:

sudo init 6

When you use init 6, all running processes are stopped and the server is restarted.

However, this approach might not always stop certain processes gracefully, making it a less ideal choice for production servers unless absolutely necessary.

For modern servers that use Systemd, the systemctl command provides more control over the reboot process.

Using the sudo systemctl reboot command ensures that systemd-managed services are stopped gracefully before the reboot, making it a safer and more reliable option for newer distributions.

Reboot via SSH

You can reboot a Linux server remotely using SSH without needing physical access.

This is especially useful in situations where the server becomes unresponsive, and a reboot is required to restore functionality.

To reboot a server via SSH, simply connect to the server using your SSH client and run the reboot command, like this:

ssh [user]@[server-ip] "sudo reboot"

Instead of using the reboot command, you could also run any other reboot command, such as shutdown or systemctl.

Reboot Through Server Provider

If you are managing a server from a server provider, like Hetzner, you should have an option to reboot the server from the cloud dashboard. Some providers offer two reboot options: a soft reboot and a hard reboot.

A soft reboot initiates a restart through the system, allowing it to safely stop services and processes, similar to running the reboot command.

In contrast, a hard reboot is essentially a forced power cycle, which can be used when the server is unresponsive and cannot be rebooted gracefully.

A hard reboot can sometimes result in data loss or file system corruption, so it should be used cautiously.

How to Reboot Safely

Rebooting a Linux server in a production environment requires careful preparation to minimize disruption.

The first and most critical step is to notify all stakeholders about the reboot. For logged-in users on the server, you can use the wall command to broadcast a system-wide message, like this:

sudo wall "Server rebooting at 4 AM."

This ensures that users are aware of the reboot and have time to save their work.

If you’re running a hosting business or managing services for clients, it’s equally important to notify customers and anyone else who could be affected by the downtime.

Next, it’s essential to verify the server’s health. Use commands like top, df -h, and free -m to check CPU, memory usage, and disk space.

Tools like htop provide a more detailed, real-time overview of the server, helping you identify any resource bottlenecks.

These checks provide an overview of the server’s performance and help identify any issues that might need attention before the reboot.

Before proceeding with the reboot, stop any critical services gracefully.

Services such as web servers (Apache or Nginx) and databases (MySQL) should be stopped manually to prevent data corruption or other issues. For example:

sudo systemctl stop nginx  
sudo systemctl stop mysql

Keep in mind, however, that some reboot commands we’ve covered – like reboot and shutdown – already stop services gracefully as part of the reboot process. If you’re using one of these commands, this step may not always be necessary, but it’s a good habit to verify and manually stop any especially critical services beforehand.

Step four is to ensure you have up-to-date backups available. Before rebooting, always verify that critical data and configurations are backed up in case anything goes wrong during the reboot process. This precaution will help you recover quickly if needed.

Finally, proceed with the reboot using your preferred method. Once the server restarts, verify that all services have come back online and are running as expected. Check the status of essential services such as Apache, MySQL, or Nginx, and inspect the logs for any errors that might have occurred during the reboot process.

Conclusion and Final Thoughts

Rebooting a Linux server, especially in production, is not a task to take lightly.

By understanding why reboots are necessary, planning thoroughly, and leveraging alternatives where possible, you can minimize downtime and ensure a smooth process.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.