Ivan Salloum

Your First Authoritative DNS Server with CoreDNS

Sat, 06 Dec 2025 00:00:00 GMT

If you’re like me, you love self-hosting your own projects. But as the number of services you run grows, so does the list of DNS records.

I used to log into my Cloudflare dashboard constantly, adding A record after A record. It felt messy, and I ended up with dozens of entries for my main domain.

I wanted a cleaner solution – a way to isolate my self-hosted projects (like Beszel) from my main site, gain full control over their DNS, and maybe even learn a thing or two about how DNS really works under the hood.

That’s where CoreDNS comes in. It’s a tiny, incredibly fast, and easy-to-configure DNS server that’s perfect for this. With a single configuration file and a lightweight Docker container, you can build your own authoritative DNS server.

In this guide, I'll walk you through the exact steps I took to set up my own authoritative DNS server for a subdomain.

We’re going to build the official source of truth for a dedicated zone, like lab.yourdomain.com. It’s a game-changer for managing a growing collection of self-hosted services.

Any public DNS query for *.lab.yourdomain.com will be correctly delegated and answered by your CoreDNS server, giving you full, instant control over your records.

Don’t worry if this is new – I'll walk you through it, step by step.

The Big Picture: What We're Building (and What We're Not)

Before we get our hands dirty, let’s zoom out.

Our goal is to build our very own authoritative DNS server. Think of it as creating a definitive, private address book for our own projects.

This guide focuses on using a subdomain (like lab.yourdomain.com) for this project, with our CoreDNS server living at ns1.lab.yourdomain.com.

This approach is known as using an "in-bailiwick" nameserver , a fancy way of saying the nameserver (ns1.lab...) resides within the same subdomain (lab...) that it manages.

It’s a clean and scalable way to structure your DNS. I chose this method to neatly separate my lab services from my main site without needing to buy a new domain, and it's a fantastic, low-cost way to learn if you already own one.

If you decide to create another DNS environment later (like dev.yourdomain.com), you can repeat the pattern without everything getting tangled up.

However, the principles here can also be adapted if you want to use a dedicated root domain (e.g., your-new-cool-project.com).

_A quick note on SEO: _A common worry is whether using a subdomain will hurt your main site’s search engine ranking. Don’t worry, it won’t. Google and other search engines treat subdomains as separate properties. As long as your projects don’t create security issues, your main domain’s SEO will be completely unaffected.

Authoritative vs. Recursive: A Crucial Distinction

It is vital to understand the type of server we are building.

A DNS server can play two main roles:

Authoritative Server (What we are building): This server holds the "master copy" of records for a specific zone. It only answers questions about domains it controls. When asked about beszel.lab.ivansalloum.com, it confidently says, "I know the answer! It's 192.168.1.100." It is the source of truth.
Recursive Resolver (What we are NOT building): This is the type of DNS server you typically use for your computer or home router (like Cloudflare's 1.1.1.1 or Google's 8.8.8.8). Its job is to go out and find the answer to any query. When you ask it for google.com, it recursively talks to other DNS servers across the internet to find the correct IP address for you.

Our configuration is purely authoritative. It knows nothing about the outside world.

If we allow our server to answer generic queries for anyone on the internet, it would become an "Open Resolver."

Hackers actively hunt for these unsecured servers to launch massive DDoS attacks (called Amplification Attacks) against victims. By keeping our server strictly Authoritative, we keep our infrastructure – and the internet – safer.

❗

Do NOT set this server as the primary DNS on your computer or router.

If you point your laptop's DNS settings only to your new CoreDNS server and then try to visit google.com, this is what happens:

Your server looks at its own Corefile.
It sees it's only responsible for lab.yourdomain.com.
It essentially replies, "I don't know who google.com is, and it's not my job to find out."

The request will fail, and for you, the internet will appear "broken."

Our server is only meant to be found by public resolvers (like Cloudflare or Google) that are looking for answers about our specific subdomain.

Preparing Your Server

Before we dive into CoreDNS specifics, we need to get your server ready.

I’m writing this guide based on my experience with an Ubuntu 24.04 LTS server hosted on a Hetzner VPS , but the steps should be broadly applicable to most Linux distributions.

👉

I run everything on Hetzner because it’s cost-effective and reliable – if you’d like to try them, here’s my referral link.

Set Your Server's Hostname

It’s good practice to give your server a meaningful hostname.

For this setup, it makes sense to use the nameserver's own FQDN (Fully Qualified Domain Name):

sudo hostnamectl set-hostname ns1.lab.ivansalloum.com

This command updates your server's hostname to reflect its role as the nameserver for your lab environment.

Remember to replace ivansalloum.com with your actual domain!

Register Your Nameserver's IP (Glue Records)

It's absolutely crucial to register the IP addresses for your nameserver with your domain registrar or DNS provider. These are often called "glue records" and they tell the internet where ns1.lab.ivansalloum.com actually lives.

Without them, no one will be able to find your custom DNS server.

You'll need to create A and optionally AAAA records for your nameserver's hostname (ns1.lab.ivansalloum.com) at your DNS provider.

Ensure these records are set up to directly point to your server's IP addresses and are not proxied if your provider offers that option (e.g., Cloudflare's orange cloud). DNS traffic for nameservers must hit your server directly.

Also, do not add an A record for just lab at this stage; we are only creating records for ns1.lab itself. The lab subdomain will be delegated later.

Free Up Port 53 (Stop `systemd-resolved`)

DNS services communicate primarily over port 53.

systemd-resolved (Ubuntu’s local DNS stub) might already be listening on this port, preventing CoreDNS from binding to it. We need to disable its stub listener.

You can check if port 53 is in use with:

sudo lsof -i :53

If you see systemd-resolved listed, follow these steps to disable it.

First, create the directory for systemd-resolved overrides:

sudo mkdir -p /etc/systemd/resolved.conf.d/

Then, set appropriate permissions for the newly created directory:

sudo chmod 755 /etc/systemd/resolved.conf.d/

Next, disable the DNSStubListener by writing the configuration to a file:

printf "[Resolve]\nDNSStubListener=no\n" | sudo tee /etc/systemd/resolved.conf.d/noresolved.conf

Finally, restart systemd-resolved to apply the changes:

sudo systemctl restart systemd-resolved.service

After restarting, sudo lsof -i :53 should no longer show systemd-resolved listening.

Configure Your Firewall (UFW)

For your CoreDNS server to receive queries, you'll need to open the necessary ports in your firewall.

DNS traffic primarily uses UDP port 53, but TCP port 53 is also essential:

sudo ufw allow 53/udp sudo ufw allow 53/tcp sudo ufw enable # If UFW is not already enabled

💡

Why both UDP and TCP? Almost all DNS queries use UDP because it's fast and lightweight. However, if a DNS response is too large, or for critical operations like zone transfers (especially important if you ever add a secondary DNS server), TCP is used.

Install Docker

We'll be running CoreDNS inside a Docker container for ease of deployment and management.

It's crucial to use up-to-date Docker Engine and Docker Compose versions because Ubuntu's default repositories often contain outdated packages.

For optimal performance and the latest features, always rely on Docker's official repositories.

👉

If you need a detailed walkthrough, I've got a dedicated guide: Installing Docker and Docker Compose on Ubuntu (Using Docker’s Official Repository).

Create CoreDNS Directories

We need a dedicated place for CoreDNS's configuration and zone files:

sudo mkdir -p /opt/coredns/{config,zones} sudo chmod -R 755 /opt/coredns

With these preparations done, your server is ready to host your authoritative CoreDNS instance!

👉

My Full Server Preparation Checklist We've just done the necessary prep for CoreDNS. If you'd like to see my complete checklist for setting up a new Ubuntu server from scratch for optimal security and performance, I've detailed everything in this guide: Preparing Your Ubuntu Server for First Use

Creating Your First Zone File

Now for the fun part. We need to create a zone file. This is a plain text file that acts as the database for your subdomain.

It contains all the DNS records – like A, AAAA, CNAME, etc. – that tell the world where to direct traffic for hosts within your zone.

First, navigate to the zones directory we created earlier:

cd /opt/coredns/zones

Next, create and open the zone file. It's a common convention to name this file db. followed by your zone name. In my case, it's db.lab.ivansalloum.com:

sudo vim db.lab.ivansalloum.com

Now, paste the following template into the file. This is the heart of your DNS setup:

$ORIGIN lab.ivansalloum.com. $TTL 3600 @ IN SOA ns1.lab.ivansalloum.com. admin.ivansalloum.com. ( 2025120601 ; Serial (YYYYMMDDnn) - CHANGE THIS ON EVERY EDIT! 7200 ; Refresh (2 hours) 120 ; Retry (2 minutes) 2419200 ; Expire (28 days) 3600 ; Negative Cache TTL (1 hour) ) IN NS ns1.lab.ivansalloum.com. ns1 IN A YOUR_SERVER_IPV4 ns1 IN AAAA YOUR_SERVER_IPV6 ; --- Your Custom Records Go Here --- beszel IN A 192.168.1.100 hestia IN A 192.168.1.1 ; --- Optional Wildcard For Instant Testing --- * IN A 192.168.0.10

Once you've pasted the template, here are the three critical edits you must make before saving:

Go through the template and replace
1. lab.ivansalloum.com.,
2. ns1.lab.ivansalloum.com.,
3. and admin.ivansalloum.com.
4. with your actual subdomain, nameserver FQDN, and admin email.
Replace YOUR_SERVER_IPV4 and YOUR_SERVER_IPV6 with the actual public IP addresses of your CoreDNS server. If you don't use IPv6, simply delete that line entirely.
The line 2025120601 is the zone's version number. It's crucial that you increment this number every time you edit the file. A common format is YYYYMMDDnn, where nn is the revision number for the day. For this first setup, I recommend setting it to today's date in YYYYMMDDnn format. This makes it easy to track your initial zone version.

Save the file and exit the editor (:wq in vim).

A Note on Zone File Naming

You might be wondering why we named our zone file db.lab.ivansalloum.com.

The db. prefix is purely for convention and readability; it's a holdover from BIND, one of the oldest DNS servers, where db stood for "database."

You could name your file records.txt if you wanted, as long as you point to it correctly in your Corefile.

Sticking to convention, however, makes your setup instantly familiar to other sysadmins (and your future self).

What Did We Just Create?

After saving your new zone file, you've essentially created the instruction manual for your lab.ivansalloum.com subdomain. In short, it does two main things:

Declares Authority: The SOA and NS records act like official stamps, announcing that your server is in charge of this zone.
Builds an Address Book: The A, AAAA, and other records map your service names (like beszel) to their IP addresses.

The most important "address book" entries are the A and AAAA records for ns1 itself. These are called glue records , and they're critical because they tell the internet how to find your nameserver in the first place.

Think of it as setting up the front desk (SOA, NS) and then filling out the directory (A, AAAA, etc.) for your new DNS office. Pretty neat, right?

Configuring CoreDNS with the `Corefile`

Our zone file is ready, holding all our DNS records.

But how do we tell CoreDNS to actually use it? This is where the Corefile comes in, and honestly, its simplicity is the main reason I fell in love with CoreDNS.

Instead of wrestling with complex, multi-file setups, CoreDNS uses one single, human-readable file to manage everything. It’s where we tell CoreDNS what zones to serve, what plugins to use, and how to behave.

Let's get it set up.

Navigate to the config directory you created earlier:

cd /opt/coredns/config

Now, create and open the Corefile itself:

sudo vim Corefile

Paste this small block of text in. This is all you need to get a basic authoritative server running:

lab.ivansalloum.com:53 { log errors file /zones/db.lab.ivansalloum.com }

Save and close the file.

This configuration is elegantly powerful. The first line, lab.ivansalloum.com:53, tells CoreDNS to listen for queries for our zone on port 53.

Inside the block, we enable two incredibly useful plugins, log and errors, which are lifesavers for debugging and seeing your server in action.

Finally, the file /zones/db.lab.ivansalloum.com line is the heart of our setup. This plugin points CoreDNS to our zone file, making it the official, authoritative source for all the records within it. The path /zones/db.lab.ivansalloum.com is relative to the inside of our future Docker container.

And that's it. We've defined our records and told CoreDNS how to serve them.

Docker Compose Configuration

Now that we have our zone file and Corefile configured, it's time to bring our CoreDNS server to life. For this, I absolutely love using Docker.

We'll use docker compose to define and run our CoreDNS service. This allows us to set up all the necessary parameters – like ports, volumes, and restart policies – in a single, easy-to-read file.

Let's create our docker-compose.yml file in the main /opt/coredns directory:

sudo vim /opt/coredns/docker-compose.yml

Paste the following content into the file:

services: coredns: image: coredns/coredns:latest container_name: coredns restart: unless-stopped ports: \- "53:53/udp" \- "53:53/tcp" volumes: \- ./config/Corefile:/Corefile:ro \- ./zones:/zones:ro

Save and close the file.

This docker-compose file defines how our CoreDNS server will run. In short, it:

Pulls the official CoreDNS image and names the container coredns.
Sets the restart policy to unless-stopped, which is a must-have for any self-hosted service to ensure it automatically starts up after a reboot.
Maps the DNS ports (53/udp and 53/tcp) from your host server to the container.
Mounts our configuration files (the Corefile and our zones directory) into the container in read-only mode, so CoreDNS can use them.

With this docker-compose.yml in place, we're just one command away from launching our very own authoritative DNS server!

Launching CoreDNS and Local Testing

We've prepared our server, crafted our zone file, and configured CoreDNS. Now, let's fire up our container and make sure everything is working as expected.

First, navigate to the main /opt/coredns directory where your docker-compose.yml file is located:

cd /opt/coredns

Now, launch CoreDNS with Docker Compose in detached mode (-d), meaning it will run in the background:

sudo docker compose up -d

You should see output indicating that the coredns container has been created and started.

With our CoreDNS server now running, let's immediately verify it's answering queries locally before we go live. This local verification step is crucial as it confirms our setup works before we expose it to the internet.

We'll use the dig command-line tool to do this. The trick is to query your own server's IP address directly , bypassing any other DNS resolvers. When we query our server directly, we're bypassing public resolvers to test our server in isolation.

💡

What's a DNS Resolver? A DNS resolver is a server (like 1.1.1.1 or 8.8.8.8) that clients (your computer) query to find the IP address of a domain name. It handles the entire process of finding the authoritative nameserver and returning the answer to you.

Remember to replace YOUR_SERVER_IPV4 with your server's actual public IPv4 address.

First, let's confirm our server is authoritative for the zone by querying the SOA record:

dig @YOUR_SERVER_IPV4 lab.ivansalloum.com SOA

You should see your SOA record returned, matching the details you put in db.lab.ivansalloum.com.

Next, let's query one of the specific test records we added, like beszel:

dig @YOUR_SERVER_IPV4 beszel.lab.ivansalloum.com

This should return the A record for beszel (e.g., 192.168.1.100).

Finally, if you included the optional wildcard record, test it:

dig @YOUR_SERVER_IPV4 anything.lab.ivansalloum.com

This should resolve to the IP address defined in your wildcard entry.

If all these dig commands return the expected answers, congratulations! Your authoritative DNS server is fully functional and ready to serve requests for your subdomain.

The next step is to tell the rest of the internet to start asking your server for these records.

We're almost there!

Delegating Your Subdomain

Our CoreDNS server is humming along locally.

The final, exhilarating step is to tell the world – specifically, through your domain registrar or DNS provider (in my case, Cloudflare) – that for your chosen subdomain, your CoreDNS server is now the one in charge. This is called delegation.

It's a powerful moment, making your self-hosted DNS server officially responsible for lab.ivansalloum.com (or whatever subdomain you chose).

Head over to your domain registrar or DNS provider and add a new NS (Nameserver) record for your subdomain with the following details::

Type : NS
Name : lab (or your chosen subdomain part, e.g., lab.ivansalloum.com)
Value : ns1.lab.ivansalloum.com. (your nameserver's FQDN, with a trailing dot!)
TTL : Auto is usually fine.

Before you save this record, there are a few crucial sanity checks to perform:

Don't Forget Your Glue Records: This entire process will only work if you've already created the A and AAAA records for ns1.lab.ivansalloum.com as we did in the "Preparing Your Server" section. These "glue records" are what allow the internet to find your nameserver in the first place.
Ensure "DNS Only" Mode: If your provider (like Cloudflare) offers a proxy service, make sure it is disabled for this NS record (e.g., a "grey cloud" in Cloudflare). The internet needs to be able to reach your nameserver directly.
Avoid Conflicting Records: Make sure you do NOT have an A record for lab itself. A subdomain can either be delegated (with an NS record) or point to an IP (with an A record), but it can't do both.

Once you've confirmed these points and saved your new NS record, you're ready for the final validation.

Public Validation: Confirming Your Delegation

The CoreDNS server is running, the zone file is configured, and your DNS provider has been updated to delegate your subdomain.

Now comes the exciting part: confirming that the entire internet sees your CoreDNS server as the authoritative source for your subdomain.

While DNS changes can sometimes take a while to propagate, subdomain delegation is usually quite fast – I've often seen it go live within 1 to 10 minutes.

Once you've given it a moment, you can verify your setup from any machine other than your CoreDNS server. This ensures you're querying external DNS resolvers, just like the rest of the world. We'll also use dig for this.

First, let's ask a public resolver (like 1.1.1.1) to confirm that our new nameserver is correctly set up for the subdomain:

dig lab.ivansalloum.com NS @1.1.1.1

You should see your nameserver (ns1.lab.ivansalloum.com.) in the answer section, which confirms the delegation is visible.

Next, let's test that your CoreDNS server is serving the actual records. Try querying one of the custom records you created in your zone file:

dig beszel.lab.ivansalloum.com @1.1.1.1

This should return the A record for beszel that you defined earlier.

If both of these commands work, your authoritative DNS server is officially live and serving records to the internet!

Managing Your DNS Records

Now that your authoritative DNS server is live, the best part is how easy it is to manage your records. The process is simple and gives you full control.

Here’s the step-by-step workflow to update or add new records.

Step 1: Edit Your Zone File

First, open your zone file:

sudo vim /opt/coredns/zones/db.lab.ivansalloum.com

Now, you can either change an existing record or add new ones.

CoreDNS supports all standard DNS record types, including A, AAAA, CNAME, TXT, and more.

For example, to add new A, CNAME, and TXT records, you would add lines similar to these:

; --- Your Custom Records Go Here --- beszel IN A 192.168.1.100 hestia IN A 192.168.1.1 wordpress IN A 192.168.1.105 ; A record for our new WordPress instance www.beszel IN CNAME beszel.lab.ivansalloum.com. ; CNAME for www.beszel pointing to beszel @ IN TXT "v=spf1 include:_spf.example.com ~all" ; TXT record for SPF

Remember to replace the example values with your own service names and data.

Step 2: Increment the Serial Number (Crucial!)

This is a critical step, primarily for secondary DNS servers (if you ever add them) to know your zone file has changed and needs updating.

The only strict rule is that the serial number must be a single number that strictly increases every time you make a change. However, using the YYYYMMDDnn format (Year, Month, Day, and a two-digit revision for changes on that day) is a widely-adopted best practice for human readability and sanity.

Here's how to manage it:

First change of the day: Update the date part to today's date and set the two-digit counter to 01. For example, if your old serial was 2025120202 and today is December 7th, 2025, your new serial would be 2025120701.
Subsequent changes on the same day: Only increment the last two digits. For example, a second change on December 7th would make it 2025120702.

Always remember to increment this number whenever you modify your zone file.

Step 3: Restart the CoreDNS Container

After saving and closing the file, you need to restart the CoreDNS container so it reloads your updated zone file.

Navigate to the directory where your docker-compose.yml file is located:

cd /opt/coredns

Then, simply run the restart command:

sudo docker compose restart

This command will gracefully restart the coredns service. It's very fast, usually taking just a second or two.

Once it restarts, CoreDNS will have loaded your new records.

Step 4: Verify Your Changes

Finally, to be sure everything worked, test your new record with dig from your local machine:

dig wordpress.lab.ivansalloum.com @1.1.1.1

You should see the new IP address (192.168.1.105) reflected in the answer.

That's it! You can repeat this process any time you need to add, update, or remove a DNS record.

Automating Zone Reloads (The Pro Way)

Manually restarting the CoreDNS container after every zone file change is simple and effective, but there’s an even better, more professional way: using the reload plugin.

This plugin watches your zone file for changes and, if it detects any, gracefully reloads the zone in the background with zero downtime. You never have to restart the container for simple record updates again!

Let's enable it.

Edit your Corefile one more time:

sudo vim /opt/coredns/config/Corefile

Add the reload plugin to the top of your server block. It should look like this:

lab.ivansalloum.com:53 { reload log errors file /zones/db.lab.ivansalloum.com }

Save the file, and then restart the container one last time to activate the reload plugin itself:

cd /opt/coredns sudo docker compose restart

From now on, you can simply edit your db.lab.ivansalloum.com file, and CoreDNS will automatically apply the changes within about 30 seconds (the default check interval).

This small change makes managing your DNS records an even more seamless experience.

❗

Don't forget to increase your serial number!

The reload plugin is smart. It only reloads the zone if it detects that the serial number has increased.

If you forget to increment the serial number, the plugin will assume nothing has changed and won't apply your updates.

So, the workflow remains:

Edit your records,
increment the serial,
and save the file.

The restart step is now automated for you.

Seeing the Difference: `dig` in Action

We've talked about the difference between an authoritative server and a recursive resolver, but let's see it live using dig.

This is the best way to truly understand the roles they play.

Query 1: A Standard Recursive Query

When you run dig for a public domain, your computer uses the default DNS resolver provided by your router or ISP:

dig ivansalloum.com

This is a recursive resolver.

Notice the flags in the header of the response: qr rd ra.

qr = Query Response
rd = Recursion Desired (You asked the resolver to find the answer for you)
ra = Recursion Available (The resolver is confirming it can do this)

The resolver went out and found the answer for you.

Query 2: Specifying a Public Resolver

We can get the same kind of recursive answer by specifying a different public resolver, like Cloudflare's 1.1.1.1:

dig ivansalloum.com @1.1.1.1

Again, you'll see the ra flag.

1.1.1.1 is a recursive resolver, and it happily found the answer.

Query 3: Asking Our Authoritative Server for a Public Domain (The "Broken" Test)

Now, let's ask our new CoreDNS server to find ivansalloum.com:

dig ivansalloum.com @YOUR_SERVER_IPV4

The result is completely different.

The status is REFUSED, and you might see a warning like recursion requested but not available.

Our server correctly refused the query because it's not a public resolver and has no idea who ivansalloum.com is.

Query 4: Asking Our Authoritative Server for a Zone It Controls

This is where our server shines.

Let's ask it for a record from the zone it's responsible for:

dig beszel.lab.ivansalloum.com @YOUR_SERVER_IPV4

This query succeeds beautifully! Now, look very closely at the flags in the header: qr aa rd.

That aa flag is the key. It stands for Authoritative Answer. This is the DNS system's way of confirming that the answer came directly from the server that is the ultimate source of truth for that domain.

You will not see this aa flag when you query for beszel.lab.ivansalloum.com using a public resolver like 1.1.1.1, because 1.1.1.1 is just relaying the answer it found from your server.

This aa flag is your proof that you have successfully built and queried a true authoritative DNS server.

Backup and Recovery with Hetzner Snapshots

Even with a stable DNS server, having a disaster recovery plan is essential for peace of mind.

For my setup on Hetzner, their snapshot feature combined with IP protection has proven to be a robust and reliable solution for disaster recovery.

The goal was to simulate a total server failure and see if I could bring my DNS server back to life from a backup.

Here's the strategy I used, combining two powerful Hetzner features.

The Strategy: Protected IPs + Snapshots

Enable IP Protection: Before anything else, I went to my Hetzner Cloud Console and enabled "IP protection" on my server's public IPv4 and IPv6 addresses. This is a critical feature that prevents your IPs from being lost if the server is deleted. It turns them into floating IPs that you can re-assign to a new server.
Take a Snapshot: With my CoreDNS server running and serving records for a test service (beszel.lab.ivansalloum.com), I took a manual snapshot of the server.

My Disaster Recovery Test

With the snapshot created and the IPs protected, it was time for the real test: I deleted the server entirely.

An interesting thing happened right after. On my home WiFi, beszel.lab.ivansalloum.com still worked for a few minutes. This is DNS caching in action! My local router or ISP's resolver had cached the record and didn't need to ask for it again.

However, when I switched to my phone's 5G network (a "fresh" resolver), the domain failed to resolve immediately. This perfectly demonstrates why DNS resilience is so important – cached records will eventually expire, and a single server outage will be felt.

Next, I provisioned a new server from the snapshot I had created. During the creation process, I re-attached my protected IPv4 and IPv6 addresses to the new instance.

The result? Within moments of the new server booting up, CoreDNS was back online.

My beszel.lab.ivansalloum.com domain started resolving correctly on all networks.

It just worked.

Automated Backups & Protection

While manual snapshots are great for point-in-time recovery, Hetzner offers additional layers of protection:

Automated Backups: For more frequent and automated restore points, consider enabling Hetzner's backup add-on. For a small additional cost (typically 20% of the server cost), this service provides daily backups that are kept for a week. A key advantage is that you can convert any automated backup into a manual snapshot before performing a restore or server deletion.
Deletion Protection: To prevent accidental deletion of your server (and thus, your DNS service), always enable deletion protection in the Hetzner Cloud Console. It's a simple toggle that can save you a lot of headache.

This comprehensive approach ensures you're well-covered for any eventuality.

Conclusion and Final Thoughts

Congratulations! You've successfully deployed your very own authoritative DNS server with CoreDNS, giving you unparalleled control over your subdomains.

This setup empowers you to:

Isolate your self-hosted projects: Keep your main domain's DNS clean and separate.
Enjoy instant control: Add, update, or remove records in seconds, without waiting for third-party providers.
Deepen your DNS understanding: You've walked through the core concepts of DNS delegation, zone files, and record management.

This journey into self-hosting your DNS is a fantastic step towards greater independence and control in your personal infrastructure.

Keep experimenting, keep learning, and enjoy the power you now have at your fingertips.

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance.

How to Run Beszel for Server Monitoring

Sun, 09 Nov 2025 00:00:00 GMT

I used to keep an eye on my Hetzner servers the old-fashioned way: SSH in, run a few commands, hop to the next box, and hope nothing was falling apart in the background.

Every monitoring tool I tried felt heavier than the services I was protecting – bloated dashboards, high CPU usage, and no way to see everything at a glance.

Beszel finally fixed that for me.

Why Beszel? (Context & Goals)

Beszel is an open-source, self-hosted server monitoring tool written in Go with a lightweight PocketBase dashboard.

It’s designed for simplicity, efficiency, and clarity - no unnecessary bloat, no agents eating your RAM.

Here’s what makes it stand out for me:

Lightweight design – The hub is a PocketBase app with a Go backend, and each server runs a tiny agent. Even with the hub and five agents, CPU usage stays under 2% on the smallest Hetzner instance.
Real metrics with real usability – The web dashboard is fast, mobile-friendly, and customizable. I can hide metrics I don’t care about (like GPU and temp), pin the ones I do, hit ⌘K to jump to any server, and even dig into Docker container stats and logs without SSH.
Alerts that replace my uptime tools – Beszel watches for downtime, CPU spikes, disks filling up, Docker containers misbehaving, and it emails me when something crosses a threshold. I don’t need separate uptime and resource monitors anymore.

Under the hood, Beszel is a hub-and-agent design. Each agent keeps a WebSocket connection to the hub, but if that ever drops (say, the proxy restarts), it automatically exposes a backup SSH tunnel on port 45876 so the hub can keep pulling metrics. That dual-path connection, plus mutual authentication and fingerprinting, makes me comfortable running it on every production server.

The rest of this guide walks through how I run Beszel in production: preparing the server and DNS, shipping the hub with Docker, securing it with Caddy, onboarding more systems, enabling health checks and alerts, and planning for disaster recovery with snapshots and backups.

Prep the Server and DNS

Before deploying Beszel, I prepare the base server – just like I do for every new self-hosted project:

Baseline hardening – I follow the "Preparing Your Ubuntu Server for First Use" guide I already published: create a non-root user, lock down SSH, enable UFW, and keep the packages up to date.
Install Docker Engine + Docker Compose – Beszel’s deployment runs on Docker, so I install Docker Engine and the Docker Compose plugin from Docker’s official repository using my guide: Installing Docker and Docker Compose on Ubuntu.
Hetzner IP protection – In the Hetzner Cloud console, I enable protection on the server and on its primary IPv4/IPv6 addresses. That way if the server gets deleted (accidentally or during a disaster), the IPs stay with me and I can reassign them later without touching DNS.

👉

I run everything on Hetzner because it’s cost-effective and reliable – if you’d like to try them, here’s my referral link.

Next comes DNS:

I create A and AAAA records for the hostname I’ll use (e.g., beszel.yourdomain.com). When I’m using Cloudflare, I keep the proxy toggled off while I’m setting up the reverse proxy – plain DNS means no cached certificates or blocked ACME requests while Caddy fetches SSL certs.

I recommend running each self-hosted project on itsown main hostname and letting that server do just one job. Beszel can technically share a box with other services, but I keep it isolated so troubleshooting stays simple.

With the base server hardened, DNS pointing to the host, Docker/Compose installed, and IP protection turned on, I’m ready to deploy the Beszel stack.

Install Beszel Hub + Local Agent with Docker

Beszel ships as two components: the hub (PocketBase + Beszel's web dashboard) and the agent that runs on every server you want to monitor.

You can install Beszel either using Docker or by using a single binary file.

It’s written in pure Go and can be easily compiled on the server using an install script that Beszel provides – but I still prefer Docker because it keeps everything in one compose file and makes backups, upgrades, and restores painless.

Beszel’s documentation shows two install flows: deploy the hub first and add agents later, or launch both hub and local agent in one go.

I prefer the second option so everything is up and monitored at the same time. That way, if I ever need to scale the hub’s resources as more agents come online, I already have baseline metrics for the hub itself.

Create the Compose File

I start by creating a working directory and moving into it:

mkdir ~/beszel cd ~/beszel

Then I create a docker-compose.yml file and drop in the official stack:

services: beszel: image: henrygd/beszel:latest container_name: beszel restart: unless-stopped ports: \- 8090:8090 volumes: \- ./beszel_data:/beszel_data \- ./beszel_socket:/beszel_socket beszel-agent: image: henrygd/beszel-agent:latest container_name: beszel-agent restart: unless-stopped network_mode: host volumes: \- ./beszel_agent_data:/var/lib/beszel-agent \- ./beszel_socket:/beszel_socket \- /var/run/docker.sock:/var/run/docker.sock:ro environment: LISTEN: /beszel_socket/beszel.sock HUB_URL: http://localhost:8090 TOKEN: KEY: ""

This is straight from Beszel’s docs – no edits beyond plugging in my token/key later.

The local agent writes metrics to /beszel_socket/beszel.sock, so the hub mounts the same path under volumes to read the data locally. Because the containers run in separate networks, the Unix socket is what lets them talk to each other without exposing anything to the outside.

The TOKEN and KEY placeholders stay put for now – we’ll grab the real values from the dashboard right after the stack starts.

With the file saved, I launch the stack:

sudo docker compose up -d

That brings both services online so I can finish wiring the local agent from the dashboard.

Wire the Local Agent in the Dashboard

Once the containers are running, I open http://<server-ip>:8090.

Beszel immediately asks me to create the first account – I type my email, pick a password, and seconds later I’m at the dashboard.

From there I add a system named Hub, copy the token/key Beszel shows, drop those values into the compose file (TOKEN/KEY), and rerun sudo docker compose up -d. Because both containers share /beszel_socket/beszel.sock, I set that socket path as the Host/IP when creating the agent so the connection lights up immediately.

You might notice the dashboard is reachable even if UFW doesn’t allow port 8090. That’s Docker at work – publishing ports through its own iptables rules (DOCKER chain) before UFW ever sees the traffic. We’ll tuck the hub behind Caddy and bind it to127.0.0.1 shortly so only HTTPS requests hit it.

Hub-Only Variant

If you’d rather bootstrap just the hub and attach agents later, you can start with a trimmed compose file:

services: beszel: image: henrygd/beszel container_name: beszel restart: unless-stopped ports: \- 8090:8090 volumes: \- ./beszel_data:/beszel_data

Notice there’s no ./beszel_socket:/beszel_socket line – that socket is only needed when a local agent is present.

Once you’re ready to monitor the hub itself, add the agent block back in and redeploy.

Secure Access with Caddy

A reverse proxy sits in front of your service, terminates HTTPS, and forwards clean traffic to the backend.

I don’t like exposing port 8090 over plain HTTP, so before I do anything serious in the dashboard I put Beszel behind Caddy.

I’ve used NGINX for years, but for single-app setups like this I switched to Caddy because it’s lightweight, handles Let’s Encrypt automatically, and takes minutes to configure.

I install Caddy from Ubuntu’s repo:

sudo apt install caddy

Then I back up the default config and overwrite /etc/caddy/Caddyfile with the snippet from Beszel’s docs:

beszel.example.com { request_body { max_size 10MB } reverse_proxy 127.0.0.1:8090 { transport http { read_timeout 360s } } }

Replace beszel.example.com with your hostname, save, and run:

sudo systemctl restart caddy

Caddy immediately requests a Let’s Encrypt certificate for that hostname and starts proxying traffic to 127.0.0.1:8090.

If ports 80/443 aren’t allowed in UFW, Caddy can’t complete the ACME challenge.

I forgot to open them the first time and hit Let’s Encrypt’s rate limit, so now I always double-check before restarting. When something looked off,sudo journalctl -u caddy --no-pager -n 100 helped me identify the issue right away.

Once Caddy is proxying to 127.0.0.1:8090, I tighten the Docker port mapping so Beszel only listens on localhost:

ports: \- 127.0.0.1:8090:8090

After updating the port mapping, I redeploy with sudo docker compose up -d.

From that point on, the dashboard is only reachable through HTTPS on my hostname.

If you’re exposing the dashboard to a whole team or running Beszel for clients, also harden the public ports that Caddy serves. My UFW+ adaptive firewall stack sits in front of ports 80 and 443, detects abusive connection spikes, and keeps the reverse proxy responsive even when lots of people are logged in.

It’s overkill for a single-user lab, but for business use it gives me the comfort that HTTPS stays protected while everyone pounds the UI.

Set `APP_URL`

Beszel recommends setting APP_URL when you’re behind a reverse proxy so notification links and agent configs use the correct URL.

Set it to the same hostname you specified in Caddy’s config. In the compose file, under the hub service, add:

environment: \- APP_URL=https://beszel.example.com

Then redeploy with sudo docker compose up -d.

Add More Servers (Systems)

Beszel calls each monitored server a “System.”

To add one, I open the dashboard, click Add System , choose the Docker option, and name it after the server’s hostname so everything stays consistent.

Beszel asks for the server’s IP and the port; I point it to the remote server I’m onboarding and leave the port at the default.

Beszel then shows a Docker compose snippet tailored for that system. I copy it to the target server, keep the same directory structure, and run sudo docker compose up -d. Because all of my servers already run Docker, I stick with the Docker agent (there’s a binary option if you prefer).

Within seconds the new agent connects and the hub starts collecting metrics – it really is that simple: click, copy, paste, deploy, done.

How Agents Stay Connected

Here’s what actually happens after you add a new system: the agent maintains two network paths, falls back automatically if the main one fails, and locks the connection down with mutual authentication.

The next few sections break that down so you know what’s happening behind the scenes.

Communication Paths (WebSocket and SSH)

Every agent maintains two ways to talk to the hub: a WebSocket that dials out to whatever you set in HUB_URL (typically your dashboard hostname) and an SSH-like tunnel on port 45876.

The WebSocket is the default; if the hub URL isn’t reachable – say, Caddy is reloading – the agent automatically spins up the SSH listener and the hub dials in to keep metrics flowing.

Port 45876 and Firewall Rules

Because the agent runs in network_mode: host so it can read the real NIC counters , Docker can’t publish port 45876 the way it does for 8090.

On each server running an agent, I open the port explicitly in UFW but only from the hub’s IPs:

sudo ufw allow in proto tcp from to any port 45876 sudo ufw allow in proto tcp from to any port 45876

That way, when the agent falls back to SSH, the hub can still reach it without exposing the port to the world.

Logs Showing the Fallback in Action

To see the switchover, I tail the agent logs:

sudo docker logs --tail=200 beszel-agent | egrep -i "connect(ed)? to hub|websocket|connected"

Example output:

2025/11/06 18:23:00 INFO WebSocket connected host=beszel.example.com 2025/11/07 09:26:17 WARN Connection closed err=EOF 2025/11/07 09:26:17 WARN Disconnected from hub 2025/11/07 09:26:17 WARN WebSocket connection failed err="unexpected status code: 502" 2025/11/07 09:26:17 INFO Starting SSH server addr=:45876 network=tcp 2025/11/07 09:26:50 INFO SSH connected addr=:56882 2025/11/07 09:26:50 INFO SSH connection established

When you see WebSocket connected, the agent is in dial-out mode.

If the WebSocket drops, the agent starts the SSH server on :45876, the hub reconnects through that path, and metrics keep flowing even if the dashboard itself is unreachable.

When the WebSocket comes back, the agent switches automatically.

Security Notes (From Beszel’s Docs)

The two communication paths are locked down even though they’re automatic:

When the hub starts, it generates an ED25519 keypair. The SSH fallback only accepts that key, offers no shell, and doesn’t execute commands. Even if the key leaked, attackers couldn’t run arbitrary commands on the agent.
Every WebSocket session starts with mutual auth: the agent sends its registration token, the hub signs a challenge, and the agent verifies it against the hub’s public key. Then the agent sends a fingerprint tied to the monitored server, and the hub makes sure it matches the original system record before letting metrics flow.

Together those safeguards keep the link private – only the original hub and the original agent can talk to each other.

Health Checks

Beszel ships with health commands for both the hub and agent.

Docker can run those commands on a schedule (healthchecks) to ask “are you OK?” and mark the container healthy or unhealthy.

The hub command is /beszel health --url http://localhost:8090, which does an HTTP GET to /api/health from inside the container and only returns success if it sees a 200 OK.
The agent command is /agent health, which verifies the agent process is running (but doesn’t guarantee the WebSocket or SSH path is live).

Healthchecks aren’t free – they run a command every time and add a bit of CPU overhead – so I balance Beszel’s “60s or more” guidance with my own experience and stick to the 60–120s range.

Docker doesn’t restart containers automatically on failure; it just updates their health status.

I use that status for two things: depends_on waits for the hub to start returning /api/health before the agent launches, and docker ps instantly shows me which container is misbehaving without digging through logs.

Before adding any healthchecks, the Hub system page in the web dashboard (or whatever you named our local agent) shows “Health: None” for both containers running on the hub server (beszel and beszel-agent). Scroll to the bottom and you’ll see the empty indicators.

On the hub server, I edit docker-compose.yml and add these blocks:

beszel: # ...existing config... healthcheck: test: ["CMD", "/beszel", "health", "--url", "http://localhost:8090"] interval: 120s timeout: 5s retries: 3 start_period: 5s beszel-agent: # ...existing config... healthcheck: test: ["CMD", "/agent", "health"] interval: 120s timeout: 5s retries: 3 start_period: 15s depends_on: beszel: condition: service_healthy

http://localhost:8090 works because it runs inside the hub container (not through Caddy). The depends_on block makes sure the local agent doesn’t start dialing until the hub is actually serving /api/health.

Beszel’s docs provide baseline healthchecks, but I tweak them slightly: start_period is 5 seconds for the hub and 15 seconds for the agent, and I add explicit timeout/retries so the status is meaningful. That depends_on block only belongs on the hub server – remote agents shouldn’t wait on anything.

After saving the file, I redeploy with sudo docker compose up -d.

From now on docker ps shows beszel ... (healthy) and beszel-agent ... (healthy), and if the hub ever stops answering 200 OK, I see unhealthy immediately.

Back in the dashboard, those “Health: None” labels are replaced with green “Healthy” badges for both containers.

If you want to add a healthcheck to a remote agent (on the server you’re monitoring), just add the agent block (without depends_on) to that system’s compose file:

healthcheck: test: ["CMD", "/agent", "health"] interval: 120s timeout: 5s retries: 3 start_period: 15s

That keeps Docker honest on each monitored server: the agent container reports healthy only if the process is running.

SMTP Configuration

If you want email alerts (server down/up, login notifications), Beszel needs an SMTP server.

In the Settings → Notifications tab you’ll see “Please configure an SMTP server.” Click it and you’re taken to the PocketBase dashboard where you can enable “Use SMTP mail server,” enter your SMTP details, and send a test email.

I use Proton Mail’s SMTP server (part of the Unlimited plan). It’s privacy-focused and reliable. If you want a free alternative, SMTP2GO is easy to set up and works well.

_I’ve used Proton products for years – if you try _Proton Mail for free and later upgrade to Unlimited , it’s worth it.

After the test email succeeds, go back to the main Beszel dashboard. Each system added has a bell icon at the right – click it to open Alert settings.

I enable the Status alert (uptime monitoring) and set it to 1 minute. Hetzner servers reboot in seconds, so I don’t get spammed during kernel reboots, but I know right away if a server goes offline.

Once SMTP is configured, Beszel also emails you when someone logs in from a new location.

PocketBase Passwords

After the initial setup (when Beszel prompts you for your first password), I immediately change the PocketBase superuser password from the hub server. This command is also your go-to if you ever lose dashboard access:

sudo docker exec beszel /beszel superuser update name@example.com newpassword

That command logs everyone out of the PocketBase admin UI.

It’s good practice to use a different password for PocketBase and Beszel, so after running the CLI command I also log into the PocketBase Users view and update the Beszel dashboard user there. Follow the same steps later whenever you want to rotate the Beszel password.

For now we do those updates separately; a single command would be nicer after an incident.

MFA OTP

Beszel also supports email-based MFA. In the hub’s compose file add:

environment: \- MFA_OTP=true

Redeploy with sudo docker compose up -d.

The next time you log out and back in, Beszel prompts for the one-time code it emails you.

This only works if SMTP is rock solid, so use a reliable provider.

Container Metrics

When I first tried Beszel, it couldn’t track Docker containers. Now it does – and even captures logs.

Open any system that runs Docker and scroll down: you’ll see each container listed with CPU/memory usage, network I/O, image, status, uptime, and a health indicator.

Above the list, Beszel plots the top CPU and memory consumers plus a Docker network I/O chart, which makes it obvious which container is spiking.

Clicking a container shows its logs (with a refresh button) and configuration details.

I also skim the beszel-agent container logs directly from the UI so I can tell if the agent fell back to SSH overnight without SSHing into the box.

CPU Metrics

The CPU charts are my favorite quick-check.

If I see a spike, I click the three dots in the CPU widget and open the CPU Time Breakdown chart.

The first thing I check is I/O wait – it’s often the culprit when disks are slow or a query is misbehaving.

Beszel also plots average utilization across cores and a per-core chart, which is great for single-threaded apps like WordPress because you can see which core is pegged.

Disaster Recovery

The final piece of the stack is disaster recovery.

Think about what happens if you misconfigure something, install a bad update, or your provider has a fire – how do you bring Beszel back online quickly?

Hetzner’s primary IP protection keeps your IPv4/IPv6 addresses even if you delete the server, but you still need a fresh server image.

Snapshots are the easiest way: take one in the Snapshots tab, and if disaster strikes you can spin up a new server from it, reassign the primary IPs, and you’re back without touching DNS or Caddy.

Snapshots are manual. If you want automated restore points, Hetzner’s backup add-on (20% of the server cost) keeps daily backups for a week.

You can convert any backup into a snapshot before deleting a server. Also enable deletion protection so you don’t nuke anything by accident.

My Recovery Checklist

Here’s the exact sequence I follow when something goes sideways:

Enable Hetzner backups and turn on deletion protection for both the server and its primary IPs.
When something goes wrong, convert the latest backup (and one or two earlier ones) into snapshots so I have multiple restore points.
Once the conversions finish, verify the snapshots exist, then disable deletion protection on the server only (leave IP protection enabled).
Delete the broken server, restore from the chosen snapshot, and assign the same primary IPv4/IPv6 addresses.
Bring Beszel back up, confirm Caddy works over HTTPS, agents reconnect, and metrics start flowing.
After everything is stable, remove extra snapshots to keep storage costs down.

I’ve tested the process: I created a snapshot, deleted the hub server, restored to a new box with the same primary IPs, and Beszel came back without any issues.

Snapshots work flawlessly for this setup.

Conclusion and Final Thoughts

You’ve seen how I run Beszel end to end: prep the server, deploy the hub + agent, lock it down behind Caddy, onboard new server (systems), keep the hub healthy with healthchecks, wire up email/MFA, and plan for disaster recovery.

The payoff is huge – I can glance at the dashboard, see which containers or cores are spiking, know when a system goes down, and have confidence I can rebuild it quickly if something goes wrong.

If you follow the same steps, you’ll have a monitoring stack that’s fast, reliable, and easy to manage.

Thanks for following along; let me know how your own setup goes.

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance.

How I Built an Adaptive Firewall Setup with UFW and Fail2ban (UFW+)

Tue, 14 Oct 2025 00:00:00 GMT

In my earlier guide, I showed how to prevent SYN flood attacks using kernel tuning , a few UFW rules , and some basic Fail2ban jails to analyze UFW’s logs and block attackers.

That setup worked – it could stop simple SYN floods – but it had limits. It was reactive, narrow in scope, and mainly focused on controlling SYN packet rates.

Since then, I’ve built something much more powerful – a project I call UFW+.

UFW+ takes the same foundation and pushes it to the next level: an adaptive firewall setup that merges UFW , Fail2ban (the log-watching tool that blocks bad actors automatically), and the performance of nftables (the modern Linux packet filtering engine) into a cohesive, modern defense layer.

By moving away from iptables and leveraging nftables’ native sets feature, it blocks abusive IPs more efficiently – without adding thousands of individual DROP rules – while Fail2ban maintains real-time ban lists directly inside nftables.

It doesn’t just rate-limit SYN packets anymore – it detects scans, spoofed traffic, and abusive connection floods in real time, then automatically bans the sources with smarter, escalating Fail2ban rules.

This guide walks you through how I built it, the reasoning behind every rule, and how you can deploy it to harden your own Linux server against modern network attacks.

Don’t worry if this stack is new – I'll walk you through each piece step by step so you can follow along at your own pace.

❗

This setup has been tested exclusively on Ubuntu 24.04 LTS. Other distributions or earlier Ubuntu versions may require adjustments.

Author’s Note

This guide – or rather, UFW+ – focuses on protecting HTTP (port 80) and HTTPS (port 443) traffic, since those are the most common public entry points.

However, the same logic and rule patterns can be applied to any service port – the principles are universal.

I plan to extend UFW+ to cover additional ports and the UDP-based version of port 443 (QUIC) in the future.

QUIC matters because most modern browsers prefer it for HTTPS traffic, so covering UDP-based port 443 keeps this setup aligned with how people actually reach your sites.

When those improvements are implemented, this guide will be updated accordingly to include the new rules and configuration examples.

It’s worth noting that while smaller attacks can often be handled with the right firewall setup, large, distributed attacks are much harder to stop completely.

In short, a Denial-of-Service (DoS) attack usually comes from one source, while a Distributed Denial-of-Service (DDoS) attack comes from many – making it far more difficult to block all of it at once.

That said, even basic protections can make your server a much harder target.

Even if your server provider (for example, Hetzner) offers some level of built-in DDoS protection, it often won’t catch the smaller, low-rate SYN floods that directly target your VPS.

That’s where UFW+ comes in – the setup you’ll learn in this guide focuses on server-side defenses that block and slow down those targeted attacks before they can cause real damage.

These layers don’t make you invincible, but they make your server resilient – and that’s the goal of everything you’ll build here.

👉

New to Hetzner? Their basic DDoS filtering and generous bandwidth make it a solid test bed – use my link to grab some free credits if you want to try it.

Background: The Old Setup and Its Limits

In my previous setup, the goal was simple: stop SYN flood attacks.

At the time, I focused on the most common issue many people were struggling with – massive bursts of half-open TCP connections that could bring a production server down almost instantly.

💡

What’s a SYN flood attack? A SYN flood is a type of DoS or DDoS attack where an attacker sends a flood of fake TCP connection requests (SYN packets) but never completes the handshake.

The old firewall setup worked: it used a few mangle table rules to drop suspicious packets, some UFW rate limits to slow down floods, and a basic Fail2ban jail that parsed UFW logs and temporarily blocked offending IPs.

That setup definitely helped – it could stop simple SYN floods and keep the server stable under light attack – but it also had limitations and design issues.

The rate limits were too strict, which sometimes blocked legitimate users. Logging wasn’t rate-limited either, so during heavy attacks the server logs could quickly fill up.

Fail2ban relied on adding one DROP rule per banned IP through iptables, which didn’t scale well when you’re dealing with hundreds of attackers.

It also became clear that SYN floods weren’t the only problem. Some attackers avoided the SYN limit altogether by opening a large number of concurrent connections and keeping them idle – a form of connection flood that slowly consumed resources.

And because I didn’t yet whitelist Cloudflare’s edge IP ranges, some Cloudflare addresses were being blocked as well , which caused legitimate traffic to get filtered out.

Overall, the old setup worked, but it had rough edges that needed to be fixed. It was time to redesign it into something more efficient, scalable, and aware of real-world traffic patterns.

Design Goals and Architecture

When I started redesigning my old setup into UFW+ , I wanted to move away from static, reactive defenses and toward an adaptive, scalable firewall setup.

The earlier approach worked for simple SYN floods, but it wasn’t sustainable under heavier or more diverse attacks.

So, I built UFW+ around three main design goals.

1. Smarter Packet Filtering at the Earliest Stage

The first step was improving packet sanity checks before traffic even reached higher firewall chains.

In the original setup, this stage only dropped invalid and non-SYN TCP packets:

-A PREROUTING -m conntrack --ctstate INVALID -j DROP -A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

That worked, but left plenty of malformed packets untouched – so I expanded it into a full set of protocol sanity checks.

The idea was to make this stage act like a first-line scrubber , eliminating garbage packets such as:

TCP packets with impossible or illegal flag combinations (like SYN+FIN or SYN+RST)
NULL and Xmas scans
Spoofed packets claiming private, loopback, or reserved IP sources
TCP handshakes with invalid MSS values, which often signal malformed or probing traffic

By handling this at the PREROUTING stage, the kernel drops malicious or malformed packets before they can consume connection-tracking resources – a lightweight way to protect the server from protocol abuse and spoofing.

2. Adaptive Connection and Rate Controls

In the earlier configuration, I used a fixed limit of 10 SYN packets per second per IP, with a small burst of 20.

There were no limits on total concurrent connections, so the firewall could only handle fast SYN floods – it didn’t recognize slower connection floods where attackers opened hundreds of idle sessions.

In short, it wasn’t adaptive. It applied the same rule to everyone, regardless of behavior, and it didn’t log efficiently either, which caused log spam during heavy attacks.

UFW+ changes that with three key improvements:

Dynamic per-source SYN rate limiting: Each source IP can open up to 30 new TCP connections per second (with a 60-packet burst). This allows for real traffic spikes while still throttling attacks.
Connection concurrency caps: To counter slow connection floods – where attackers open many idle sessions instead of rapid SYN bursts – each IP is capped at 100 concurrent connections.
Rate-limited logging (1 log per minute per IP): Both the connection-limit and SYN flood rules log at most once per minute per source. This prevents log flooding under attack and gives Fail2ban a clean, steady signal to react to.

These controls are layered: connection limits are evaluated first , then SYN flood checks apply only to new handshakes that pass through.

Cloudflare edge IPs are whitelisted early so legitimate proxied traffic bypasses the analysis entirely.

3. Fail2ban as an Adaptive Enforcement Layer

The biggest shift in UFW+ is how bans are applied and managed.

Previously, I used the classic iptables action in Fail2ban:

action = iptables[type=allports, name=synflood, chain=fail2ban, protocol=tcp]

Each ban inserted a new DROP rule directly into the firewall – fine for a few attackers, but not for hundreds.

Instead of inserting one DROP rule per IP (which doesn’t scale well in iptables), Fail2ban now interacts directly with nftables sets – a native and efficient way to maintain large lists of banned IPs.

Each ban action simply adds the offending IP address to an nftables set referenced by a single rule, keeping the ruleset compact and performant.

The Fail2ban configuration itself became smarter and more dynamic:

It now uses the systemd journal backend to read kernel-level UFW logs directly, improving accuracy and avoiding duplicate log parsing.
Two specialized jails monitor different log patterns:
- synflood for excessive handshake attempts
- connlimit for clients exceeding concurrent connection caps
Both jails share the same adaptive banning policy : 3 strikes within 5 minutes triggers a 5-minute ban, which doubles on each repeat offense (up to 1 hour). This gives real users a margin of error while escalating punishment for persistent attackers.
A recidive jail tracks repeat offenders across days, applying bans that grow up to 3 weeks for chronic abuse.

✅

The Fail2ban database retention was extended to 30 days to support long-term tracking for recidive detection.

Architecture Overview

To understand how all the pieces of UFW+ fit together, it helps to look at the overall flow of packet handling and rule evaluation.

Because nftables processes packets based on hook priorities (lower numbers are evaluated first), Fail2ban’s drop rule chain is inserted with a priority such that banned IPs are dropped before any UFW-based chain.

Think of it like airport security: the ID check happens before the boarding gate, so someone flagged early never reaches the later stations.

In practice, this happens even before the before.rules file is evaluated, ensuring that once an IP is banned, it never reaches our UFW logic at all.

Here’s how the full traffic flow works:

┌───────────────────────────────────────────┐ │ Kernel (nftables mangle) │ │ → Drops invalid/spoofed/malformed packets │ └───────────────────────────────────────────┘ ↓ ┌───────────────────────────────────────────┐ │ Fail2ban nftables drop sets │ │ → Banned IPs are dropped immediately │ │ via high-priority hook (before UFW) │ └───────────────────────────────────────────┘ ↓ ┌───────────────────────────────────────────┐ │ UFW (nftables backend) │ │ → Cloudflare whitelist (first) │ │ → Per-IP connection concurrency limits │ │ → SYN flood rate limiting │ └───────────────────────────────────────────┘ ↓ ┌───────────────────────────────────────────┐ │ Fail2ban adaptive layer │ │ → Monitors kernel logs (systemd backend) │ │ → Updates nftables ban sets in real time │ │ → Escalates bans on repeat offenses │ └───────────────────────────────────────────┘

This order ensures that:

Cloudflare traffic is trusted and bypasses analysis entirely.
Connection caps are evaluated before SYN flood checks (since they act on already-open sessions).
SYN flood limits handle new connection bursts.
Fail2ban bans cut off attackers before UFW even runs.

Everything above the UFW layer (the mangle table and Fail2ban drop chain) happens early in packet processing, which keeps the firewall fast and efficient even under heavy attack.

Configuration Walkthrough

Now that you understand the reasoning and design behind UFW+ , it’s time to put it into action.

This walkthrough takes you through every step of building the setup – from preparing your server and defining early packet filters to configuring adaptive UFW and Fail2ban rules and integrating them with nftables.

You can follow along on a fresh Ubuntu 24.04 LTS server (recommended), or adapt the steps for another Debian-based distribution.

Each section includes both the commands and the reasoning behind them – so you’ll know why every rule exists, not just how to copy it.

Server Preparation

Before adding any firewall rules, it’s important to start with a clean and secure foundation.

These steps prepare your Ubuntu 24.04 LTS server for the UFW+ setup, ensuring consistency, proper logging, and a safe configuration environment.

Update and upgrade your server

Always begin by bringing your server up to date.

This ensures you have the latest kernel, security patches, and package versions:

sudo apt update && sudo apt dist-upgrade -y

If the server reports that a reboot is required to apply a new kernel, you can reboot later after completing all preparation steps.

Create a non-root user

As a good security practice, avoid working directly as root.

Instead, create a new user with sudo privileges and use it for all management tasks:

sudo adduser yourusername sudo usermod -aG sudo yourusername

Then exit the SSH session and reconnect using your new user.

Set your hostname and timezone

Setting a proper hostname and timezone helps keep your logs consistent and easy to read:

sudo hostnamectl set-hostname yourservername.yourdomain.com sudo timedatectl set-timezone Your/Timezone

Always set a valid FQDN (Fully Qualified Domain Name) for the hostname.

Enable UFW and protect SSH access

Start by enabling UFW and applying a rate limit to your SSH port:

sudo ufw limit 22/tcp sudo ufw enable

This allows SSH connections but slows down repeated failed attempts, reducing the chance of brute-force attacks.

Allow HTTP and HTTPS traffic

Next, open your web ports (80 and 443) – these should always be accessible to the public:

sudo ufw allow 80/tcp sudo ufw allow 443/tcp

At this stage, you’re simply allowing web traffic – UFW+ will later analyze and protect these ports.

Install a web server for testing (optional)

If you’re working on a clean test server, install Nginx to generate normal web traffic for testing:

sudo apt install nginx -y

If you’re running this setup on a production server, you can use your existing web server instead.

Install and prepare Fail2ban

Finally, install Fail2ban – it will serve as the adaptive banning layer in your UFW+ setup:

sudo apt install fail2ban -y

Then copy the default configuration files to their editable .local versions:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local sudo cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local

Editing .local files ensures that your custom settings won’t be overwritten during server or package updates.

Reboot your server

Once everything is done, reboot the server:

sudo reboot

This ensures that all changes are properly applied and any pending updates or configurations take full effect.

👉

For a complete walkthrough on preparing a fresh server – including disabling root access, using SSH keys, and other best practices – check out my full guide: Preparing Your Ubuntu Server for First Use

UFW and nftables on Modern Ubuntu

Starting with Ubuntu 22.04 and later (including 24.04) , UFW uses nftables under the hood by default – even though it still looks like it’s using iptables.

When you run UFW commands, they’re translated through the iptables-nft compatibility layer, which means the actual packet filtering happens inside nftables.

You can confirm this by running:

sudo update-alternatives --display iptables

And you’ll see something like:

link currently points to /usr/sbin/iptables-nft

To view the live nftables rules created by UFW, run:

sudo nft list ruleset

In short, UFW on modern Ubuntu already uses nftables by default – UFW+ simply takes advantage of that.

Base Mangle Table Rules (Packet Sanity Filtering)

The first real line of defense in UFW+ starts at the mangle table – before UFW’s normal chains even run.

The mangle table handles packets very early – in the PREROUTING stage – before they reach normal firewall rules.

These rules act as packet sanity checks , filtering out clearly invalid, spoofed, or malformed packets as early as possible to prevent them from consuming server resources.

Edit the UFW `before.rules` file (IPv4)

Open the file using your preferred text editor:

sudo vim /etc/ufw/before.rules

I use vim, but you can use nano, vi, or anything you like.

💡

If you’re using Vim: Press i to enter insert mode and make your edits. When you’re done, press Esc, then type :wq and hit Enter to save and exit.

Scroll to the very bottom of the file (after the last COMMIT line), and append the following block:

*mangle :PREROUTING ACCEPT [0:0] # Drop all packets flagged as INVALID by connection tracking -A PREROUTING -m conntrack --ctstate INVALID -j DROP # Drop NEW incoming TCP packets that do not have the SYN flag (possible scans/attacks) -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP # Drop packets with all TCP flags set (Xmas tree scan) -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP # Drop packets with no TCP flags set (NULL scan) -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP # Drop TCP packets with illegal flag combinations -A PREROUTING -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP # Drop packets with invalid or spoofed source addresses -A PREROUTING -s 0.0.0.0/8 -j DROP -A PREROUTING -s 10.0.0.0/8 -j DROP -A PREROUTING -s 172.16.0.0/12 -j DROP -A PREROUTING -s 192.168.0.0/16 -j DROP -A PREROUTING -s 127.0.0.0/8 ! -i lo -j DROP # Drop TCP packets with an invalid MSS (Maximum Segment Size) -A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP COMMIT

Save and exit.

Edit the UFW `before6.rules` file (IPv6)

Open:

sudo vim /etc/ufw/before6.rules

Scroll to the bottom (again, after the last COMMIT line), and append this block:

*mangle :PREROUTING ACCEPT [0:0] # Drop all packets flagged as INVALID by connection tracking -A PREROUTING -m conntrack --ctstate INVALID -j DROP # Drop NEW incoming TCP packets that do not have the SYN flag (possible scans/attacks) -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP # Drop packets with all TCP flags set (Xmas tree scan) -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP # Drop packets with no TCP flags set (NULL scan) -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP # Drop TCP packets with illegal flag combinations -A PREROUTING -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP # Drop IPv6 ULA (Unique Local Addresses) and spoofed loopback -A PREROUTING -s fc00::/7 -j DROP -A PREROUTING ! -i lo -s ::1/128 -j DROP # Drop TCP packets with an invalid MSS (Maximum Segment Size) -A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 1220:65535 -j DROP COMMIT

Save and exit.

Reload UFW

Apply the changes:

sudo ufw reload

This reloads UFW and applies the new mangle table rules immediately – there’s no need to reboot or restart any services.

Verify the new rules

To confirm that your new mangle table rules are active:

sudo nft list ruleset | grep mangle -A 20

You should see your PREROUTING entries under the mangle table.

What these rules do

Each rule in the mangle table serves a specific purpose.

Together, they form a packet sanity layer that keeps invalid or malicious traffic from even entering your firewall.

Drop `INVALID` packets

These are packets flagged by the Linux connection tracker as inconsistent – for example:

A reply to a connection that doesn’t exist,
Fragments missing from a previous packet, or
Packets with mismatched headers.

They’re useless for normal communication and often appear during floods or malformed traffic bursts.

Drop `NEW` TCP packets without the SYN flag

A valid TCP connection always starts with a SYN packet.

If a packet claims to start a new connection (NEW state) but has no SYN flag , it’s almost certainly part of a scan or spoofed attempt.

Block Xmas and NULL scans

Attackers sometimes send packets with:

All flags set (Xmas tree scan), or
No flags at all (NULL scan).

These are classic reconnaissance techniques – often used by tools like Nmap – to map open ports or trigger unexpected responses.

They’re never part of normal TCP traffic.

Drop illegal TCP flag combinations

Packets that combine SYN+FIN or SYN+RST flags violate the TCP standard – a connection can’t be both starting and finishing (or resetting) at once.

They often indicate broken or malicious network stacks.

Drop spoofed source IP addresses

These rules block packets pretending to come from:

Private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
Reserved addresses (0.0.0.0/8)
Localhost spoofing (127.0.0.0/8 arriving on a non-loopback interface)

Traffic from these ranges should never appear on a public WAN interface.

If it does, it’s almost always a spoofed packet – someone faking source IPs to hide their identity or poison routing tables.

Drop packets with invalid MSS values

This rule checks the Maximum Segment Size (MSS) in new TCP handshakes and drops any packet outside the legal range:

536 bytes – minimum allowed for IPv4 (RFC 879)
1220 bytes – minimum derived for IPv6 (based on RFC 8200's 1280 Bytes MTU)
65535 bytes – theoretical maximum

If a SYN packet advertises a value smaller or larger than these limits, it’s likely malformed or intentionally crafted.

Whitelisting Cloudflare IP Ranges

If your website is behind Cloudflare , most incoming connections to your server actually come from Cloudflare’s edge proxy IPs , not directly from your visitors.

To avoid rate-limiting or blocking Cloudflare itself, you need to whitelist their IP ranges before your other firewall chains (like connection or SYN flood limits) are evaluated.

This ensures that all legitimate traffic passed through Cloudflare reaches your server without restriction.

Edit the UFW `before.rules` file (IPv4)

Open the file:

sudo vim /etc/ufw/before.rules

Scroll until just before UFW’s final # don't delete the 'COMMIT' line in the *filter section, and insert the following block:

:CF-EDGE - [0:0] # Send only web traffic to the CF-EDGE chain -A ufw-before-input -p tcp --dport 80 -j CF-EDGE -A ufw-before-input -p tcp --dport 443 -j CF-EDGE # Allow from Cloudflare edge networks -A CF-EDGE -s 173.245.48.0/20 -j ACCEPT -A CF-EDGE -s 103.21.244.0/22 -j ACCEPT -A CF-EDGE -s 103.22.200.0/22 -j ACCEPT -A CF-EDGE -s 103.31.4.0/22 -j ACCEPT -A CF-EDGE -s 141.101.64.0/18 -j ACCEPT -A CF-EDGE -s 108.162.192.0/18 -j ACCEPT -A CF-EDGE -s 190.93.240.0/20 -j ACCEPT -A CF-EDGE -s 188.114.96.0/20 -j ACCEPT -A CF-EDGE -s 197.234.240.0/22 -j ACCEPT -A CF-EDGE -s 198.41.128.0/17 -j ACCEPT -A CF-EDGE -s 162.158.0.0/15 -j ACCEPT -A CF-EDGE -s 104.16.0.0/13 -j ACCEPT -A CF-EDGE -s 104.24.0.0/14 -j ACCEPT -A CF-EDGE -s 172.64.0.0/13 -j ACCEPT -A CF-EDGE -s 131.0.72.0/22 -j ACCEPT # Anything else continues through normal UFW processing -A CF-EDGE -j RETURN

Then leave UFW’s own COMMIT line in place, and below it you’ll still have your *mangle section and its own COMMIT – keep both.

These rules redirect web traffic (ports 80 and 443) from the main ufw-before-input chain into our custom CF-EDGE chain. That allows UFW to quickly check if a packet comes from one of Cloudflare’s edge IPs.

If it does, it’s accepted immediately. If not, it continues through the normal UFW rule flow.

Edit the UFW `before6.rules` file (IPv6)

Open the file:

sudo vim /etc/ufw/before6.rules

Add this block before the final COMMIT in the *filter section:

:CF6-EDGE - [0:0] # Send only web traffic to the CF6-EDGE chain -A ufw6-before-input -p tcp --dport 80 -j CF6-EDGE -A ufw6-before-input -p tcp --dport 443 -j CF6-EDGE # Allow from Cloudflare edge networks -A CF6-EDGE -s 2400:cb00::/32 -j ACCEPT -A CF6-EDGE -s 2606:4700::/32 -j ACCEPT -A CF6-EDGE -s 2803:f800::/32 -j ACCEPT -A CF6-EDGE -s 2405:b500::/32 -j ACCEPT -A CF6-EDGE -s 2405:8100::/32 -j ACCEPT -A CF6-EDGE -s 2a06:98c0::/29 -j ACCEPT -A CF6-EDGE -s 2c0f:f248::/32 -j ACCEPT # Anything else continues through normal UFW processing -A CF6-EDGE -j RETURN

These rules do the same as the IPv4 ones – they allow Cloudflare’s IPv6 edge IPs to pass through without restriction.

Reload UFW

Apply the changes:

sudo ufw reload

This applies your updated rules immediately.

❗

Note: Cloudflare’s IP ranges sometimes change. You can always find the latest list here: https://www.cloudflare.com/ips/ Consider writing a small script or cron job that checks this list regularly and updates your rules automatically when Cloudflare adds or removes IP ranges.

Connection Limits & SYN Flood Protection

After filtering packets and whitelisting Cloudflare, it’s time to add the connection and rate control rules.

These will become the core of your firewall – keeping traffic balanced and preventing abusive patterns from overwhelming the server.

Edit the UFW `before.rules` file (IPv4)

Open the file:

sudo vim /etc/ufw/before.rules

Then add these blocks after your Cloudflare rules and before UFW’s final COMMIT line:

:CONNLIMIT - [0:0] # Send only new SYN packets for web ports to the CONNLIMIT chain -A ufw-before-input -p tcp --syn --dport 80 -j CONNLIMIT -A ufw-before-input -p tcp --syn --dport 443 -j CONNLIMIT # connlimit checks how many concurrent connections each source IP already holds # If a single IP opens more than 100 concurrent connections: # - Log the event (limited to once per minute to avoid log spam) # - Drop further connections -A CONNLIMIT -m conntrack --ctstate NEW -m connlimit --connlimit-saddr --connlimit-above 100 --connlimit-mask 32 -m limit --limit 1/min --limit-burst 1 -j LOG --log-prefix "[UFW Connlimit] " -A CONNLIMIT -m conntrack --ctstate NEW -m connlimit --connlimit-saddr --connlimit-above 100 --connlimit-mask 32 -j DROP # Below the limit → continue to SYN_FLOOD_PROTECTION -A CONNLIMIT -j RETURN :SYN_FLOOD_PROTECTION - [0:0] # Send only new SYN packets for web ports to the SYN_FLOOD_PROTECTION chain -A ufw-before-input -p tcp --syn --dport 80 -j SYN_FLOOD_PROTECTION -A ufw-before-input -p tcp --syn --dport 443 -j SYN_FLOOD_PROTECTION # Normal traffic (≤30 new connections/sec, 60 burst) → allow immediately # hashlimit keeps track of per-IP connection rates so we can allow normal visitors -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit4 --hashlimit-upto 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 32 -j RETURN # If rate exceeded → log once per minute, then drop -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit4 --hashlimit-above 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 32 -m limit --limit 1/minute --limit-burst 1 -j LOG --log-prefix "[UFW SYN Flood Detected] " -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit4 --hashlimit-above 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP # Return to normal UFW processing -A SYN_FLOOD_PROTECTION -j RETURN

Let’s quickly break down what’s actually happening behind these rules.

They use a few important firewall modules that make rate control possible:

connlimit – counts how many concurrent connections each IP address has open.
hashlimit – tracks packet rates per IP address. It limits how many new TCP handshakes an IP can start each second.
limit – rate-limits how often a rule can log messages. This keeps your logs clean and prevents a flood of entries during an attack.

Together, these modules make the firewall adaptive – it reacts to each IP individually instead of applying one static limit to everyone.

Edit the UFW `before6.rules` file (IPv6)

Open the file:

sudo vim /etc/ufw/before6.rules

Add the equivalent IPv6 version:

:CONNLIMIT - [0:0] # Route new SYN packets on web ports to the CONNLIMIT chain -A ufw6-before-input -p tcp --syn --dport 80 -j CONNLIMIT -A ufw6-before-input -p tcp --syn --dport 443 -j CONNLIMIT # connlimit checks how many concurrent connections each IPv6 address holds # If a single IPv6 address opens more than 100 concurrent connections: # - Log once per minute # - Drop further connections -A CONNLIMIT -m conntrack --ctstate NEW -m connlimit --connlimit-saddr --connlimit-above 100 --connlimit-mask 128 -m limit --limit 1/min --limit-burst 1 -j LOG --log-prefix "[UFW Connlimit] " -A CONNLIMIT -m conntrack --ctstate NEW -m connlimit --connlimit-saddr --connlimit-above 100 --connlimit-mask 128 -j DROP # Below the limit → continue to SYN_FLOOD_PROTECTION -A CONNLIMIT -j RETURN :SYN_FLOOD_PROTECTION - [0:0] # Route new SYN packets on web ports to the SYN_FLOOD_PROTECTION chain -A ufw6-before-input -p tcp --syn --dport 80 -j SYN_FLOOD_PROTECTION -A ufw6-before-input -p tcp --syn --dport 443 -j SYN_FLOOD_PROTECTION # Normal traffic (≤30 new connections/sec, 60 burst) → allow immediately # Same hashlimit module handles IPv6 sources, just with a /128 mask # hashlimit keeps track of per-IP connection rates so we can allow normal visitors -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit6 --hashlimit-upto 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 128 -j RETURN # Over the rate limit → log once per minute, then drop -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit6 --hashlimit-above 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 128 -m limit --limit 1/minute --limit-burst 1 -j LOG --log-prefix "[UFW SYN Flood Detected] " -A SYN_FLOOD_PROTECTION -m conntrack --ctstate NEW -m hashlimit --hashlimit-name synlimit6 --hashlimit-above 30/second --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-srcmask 128 -j DROP # Return to normal UFW processing -A SYN_FLOOD_PROTECTION -j RETURN

The IPv6 rules work exactly the same way, using the same modules and logic – the only difference is the address mask :

IPv4 uses /32 (one full address).
IPv6 uses /128 to match a single host, since IPv6 addresses are much longer.

Other than that, both rule sets behave identically: each monitors connections and SYN rates per IP, logs suspicious activity once per minute, and drops abusive traffic early.

Reload UFW

Apply the changes:

sudo ufw reload

This applies your updated rules immediately.

How it works

The two custom chains – CONNLIMIT and SYN_FLOOD_PROTECTION – each handle a different kind of abuse pattern:

CONNLIMIT: Prevents slow connection floods – when an attacker opens many concurrent connections and leaves them idle to consume memory and sockets. If a single IP holds more than 100 active connections at once, it’s logged (once per minute) and then dropped. All other IPs pass through normally.
SYN_FLOOD_PROTECTION: Prevents fast SYN floods – large bursts of new TCP handshake attempts designed to overwhelm the kernel’s connection table. Allows up to 30 new SYN handshakes per second (with a burst buffer of 60). Anything above that is logged once per minute and dropped.

The two chains complement each other by handling different time scales of abuse:

CONNLIMIT stops long-lived, slow-drip floods (hundreds of half-open or idle sessions).
SYN_FLOOD_PROTECTION stops short, intense bursts of new handshakes.

Both chains work quietly in the background.

Tuning the limits

You can safely adjust these numbers depending on your traffic patterns:

Increase --connlimit-above 100 if your app legitimately uses many concurrent connections (for example, websockets or APIs).
Adjust --hashlimit-upto 30/second and --hashlimit-burst 60 if your users trigger false positives under heavy load (for example, during peak visits).

Start conservative and monitor logs to see what works best for your environment.

Fail2ban Integration with nftables

With our UFW rules in place, it’s time to make the setup adaptive – this is where Fail2ban comes in.

Fail2ban watches for repeated bad behavior in your logs and automatically blocks those IPs using nftables sets.

Think of it as the memory layer of your firewall – it notices patterns and reacts.

Disable the Default SSH Jail (Optional)

On Ubuntu, Fail2ban comes with the SSH jail enabled out of the box.

You can confirm this by running:

sudo fail2ban-client status

You’ll see the sshd jail already running. That’s controlled by this file:

/etc/fail2ban/jail.d/defaults-debian.conf

It includes:

[sshd] enabled = true

Personally, I don’t need this one – because SSH is already limited by UFW (sudo ufw limit 22/tcp), and I only use SSH keys with root login and password authentication disabled.

If you’re in the same situation (which is recommended), you can safely disable the jail. Just don’t edit the default file – create a small override instead:

sudo vim /etc/fail2ban/jail.d/override.local

Add:

[sshd] enabled = false

That’s cleaner and future-proof – your settings won’t be overwritten by package updates.

Create Filters for UFW Log Events

Next, we’ll teach Fail2ban how to recognize our custom UFW log messages.

We’ll create two filters: one for SYN flood logs and one for connection limit logs.

Go to the /etc/fail2ban/filter.d/ directory and create two new filter files:

cd /etc/fail2ban/filter.d sudo touch synflood.conf connlimit.conf

Open synflood.conf and add:

[Definition] failregex = .*UFW SYN Flood Detected.*SRC=.*DPT=\d+.* ignoreregex =

Then open connlimit.conf and add:

[Definition] failregex = .*UFW Connlimit.*SRC=.*DPT=\d+.* ignoreregex =

That’s it.

Now whenever UFW logs a [UFW SYN Flood Detected] or [UFW Connlimit] message, Fail2ban will now recognize it instantly and take action.

Extend Database Retention

By default, Fail2ban forgets everything after a day. That’s not ideal for catching repeat offenders

For long-term tracking (especially for the recidive jail), we extend Fail2ban’s database retention period to 30 days.

Open the /etc/fail2ban/fail2ban.local file and update:

dbpurgeage = 30d

This makes sure Fail2ban remembers old bans, which is essential for the recidive jail we’ll set up next.

Create the Jails

Now for the main part – the jails that actually watch our logs and block bad IPs.

Go to the /etc/fail2ban/jail.d/ directory and create the files:

cd /etc/fail2ban/jail.d sudo touch synflood.local connlimit.local recidive.local

Open synflood.local and add:

[synflood] enabled = true backend = systemd journalmatch = _TRANSPORT=kernel usedns = no filter = synflood banaction = nftables[type=allports, blocktype=drop] maxretry = 3 findtime = 5m bantime = 5m bantime.increment = true bantime.factor = 2 bantime.maxtime = 1h bantime.rndtime = 1m

Then open connlimit.local and add:

[connlimit] enabled = true backend = systemd journalmatch = _TRANSPORT=kernel usedns = no filter = connlimit banaction = nftables[type=allports, blocktype=drop] maxretry = 3 findtime = 5m bantime = 5m bantime.increment = true bantime.factor = 2 bantime.maxtime = 1h bantime.rndtime = 1m

Finally, open recidive.local and add:

[recidive] enabled = true usedns = no filter = recidive banaction = nftables[type=allports, blocktype=drop] logpath = /var/log/fail2ban.log maxretry = 3 findtime = 1d bantime = 7d bantime.increment = true bantime.factor = 2 bantime.maxtime = 21d bantime.rndtime = 1m

These three jails work together to handle short-term attacks and long-term offenders.

Each listens for the UFW log messages we defined earlier and bans abusive IPs automatically using nftables sets – fast, clean, and scalable even with hundreds (or thousands) of bans.

How the Jails Work

Now that the jails are in place, here’s what’s actually happening behind the scenes.

Each jail watches for specific log messages coming from UFW.

When one of your firewall rules logs an event – like [UFW SYN Flood Detected] or [UFW Connlimit] – Fail2ban picks it up, checks how often it happens for that IP, and decides what to do next.

If the same IP keeps showing up too frequently within a certain time window (findtime), Fail2ban considers it abusive and bans it for the amount of time defined by bantime.

Why these settings:

backend = systemd with** journalmatch = _TRANSPORT=kernel** – tells Fail2ban to read the systemd journal directly instead of scanning text logs. This is faster and more reliable on modern Ubuntu servers.
usedns = no – skips reverse DNS lookups to keep bans fast and predictable.
banaction = nftables[type=allports, blocktype=drop] – drops offenders through an nftables set so one rule blocks every port.
**maxretry = 3** – three strikes inside findtime triggers a ban.
findtime = 5m (or 1d for recidive) – the window Fail2ban uses to count those strikes. For example, if an IP triggers the rule three times in five minutes, it’s banned.
bantime = 5m – short first ban so real users recover quickly from mistakes.
bantime.increment = true + bantime.factor = 2 – makes bans smarter. If an IP keeps getting caught again, its ban time doubles each time (up to bantime.maxtime).
bantime.rndtime = 1m – adds a random offset so bans expire at slightly different times instead of all at once.

The recidive jail functions as long-term memory – monitoring Fail2ban’s own logs to identify IPs that have been banned multiple times across days or weeks.

Apply the Changes

Once everything’s ready, restart Fail2ban:

sudo systemctl restart fail2ban sudo fail2ban-client status

You should now see your three new jails (synflood, connlimit, and recidive) active and running.

Why I Use `DROP` (silent) instead of `REJECT` / `TCP RST`

I decided early on that I wanted this setup to be quiet. No messages, no hints, no you’re blocked replies – just silence.

That’s why every rule in UFW+ uses DROP, not REJECT or REJECT with tcp-reset.

When you DROP a packet, it just disappears. The sender gets no answer at all.

When you REJECT it, you actually send something back – an error or reset – which tells whoever’s on the other side that you’re alive and blocking them.

For normal users, that kind of feedback is nice and clean. But for attackers, it’s valuable information.

A fast connection refused or port unreachable tells their scanner exactly what’s happening – and that’s the opposite of what I want.

With DROP, everything goes dark. Scanners hang, tools time out, and automated floods get slower because they have no signal to adjust their timing or rate.

Sure, it’s not perfect – some legitimate tools might just sit there waiting until they time out – but for a public-facing web server, that tradeoff is totally worth it.

So, I use DROP because I want the firewall to act like a black hole. If someone’s attacking my server, I don’t want them to know they hit a wall – I want them to wonder if they ever even reached me.

How It All Flows Together

Now that everything’s in place – the mangle table filters, Cloudflare whitelist, firewall rules, and Fail2ban jails – it’s worth seeing how the whole setup actually behaves.

Think of this as a quick packet journey through UFW+.

Normal Visitor (Direct Access)

Mangle (PREROUTING) drops any junk or malformed packets.
Passes to Fail2ban (no ban, clean IP).
Reaches CF-EDGE → returns normally.
Hits CONNLIMIT → under 100 connections, returns.
Hits SYN_FLOOD_PROTECTION → under 30 SYNs/sec, returns.
Finally, it reaches your web server – fast and clean.

Cloudflare Traffic

Same initial mangle and Fail2ban checks.
Hits CF-EDGE and gets accepted** immediately**.** It skips the CONNLIMIT and SYN_FLOOD_PROTECTION limit stages entirely (by design).

Slow Connection Hoarder

Opens hundreds of idle sockets – say, 150 at once.
Passes the SYN rate limit (only 5 SYNs/sec).
Hits CONNLIMIT → logs the event, drops anything over 100 connections. That IP starts failing silently after its 100th connection.
After repeated detections, Fail2ban steps in, adds the IP to the nftables ban set, and future packets die immediately.

SYN Flood (Fast Burst)

Sends a wave – say 200 SYN packets per second.
Trips the SYN_FLOOD_PROTECTION hashlimit.
Logs once per minute and drops the rest (silent).
After repeated detections, Fail2ban steps in, adds the IP to the nftables ban set, and future packets die immediately.

Kernel Hardening (Complementary to UFW+)

Your firewall takes care of the traffic that tries to reach your server – but what about the packets that actually make it through?

You can go one step further by hardening the kernel itself , so that even if bad packets slip through, the server’s networking behavior remains safe and predictable.

UFW+ filters the noise: it drops invalid, spoofed, or suspicious packets before they ever touch your services.
Kernel hardening (via sysctl) tells the kernel how to behave – what to ignore, what to drop, and how to react when things look weird at the protocol level.

Together, they make a perfect pair: your firewall cleans the input, and your kernel stays disciplined about what it accepts.

Adding Kernel Hardening Settings

UFW's /etc/ufw/sysctl.conf already includes a few sensible defaults – for example, it disables ICMP redirects and ignores bogus ICMP errors.

You’ll simply add extra security parameters below them to enhance protection further.

Open the file:

sudo vim /etc/ufw/sysctl.conf

Append these settings at the end:

# Enable SYN cookies (mitigates SYN floods at TCP level) net.ipv4.tcp_syncookies = 1 # Drop invalid packets at routing level net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 # Don’t accept source-routed packets net.ipv4.conf.all.accept_source_route = 0 net.ipv6.conf.all.accept_source_route = 0 # Protect against TIME-WAIT assassination net.ipv4.tcp_rfc1337 = 1

Save and apply the changes:

sudo ufw reload

That’s it.

UFW automatically loads its own sysctl file on startup, so these rules will take effect right away – no separate sysctl -p needed.

Keeping your hardening settings here also keeps everything tidy – firewall logic and kernel behavior in one place, under UFW’s control.

👉

Want to go deeper? For a full breakdown of Linux kernel hardening – including detailed explanations and additional sysctl options – check out my complete guide: Kernel Hardening: Securing Your Linux Server

Testing and Verification

I test UFW+ with two Hetzner VMs: one runs the UFW+ setup (the victim) and the other is my attacker box with apache-utils installed so I can use the ab command.

Below I show the exact commands I ran, the logs I saw, and what each result means.

👉

New to Hetzner? Use my link to get free credits!

Quick SYN-Flood Smoke Test

From the attacker I run this command, which opens a lot of new connections quickly (no keep-alive):

sudo ab -n 500 -c 20 http://victim-ipv4/

I watch the logs on the victim in real time:

sudo tail -f /var/log/syslog | grep -E 'UFW Connlimit|UFW SYN Flood Detected'

What happened to me:

The SYN_FLOOD_PROTECTION chain started logging [UFW SYN Flood Detected] messages.
After three log entries inside five minutes, Fail2ban banned the attacker IP and added it to an nftables set.

I verified the ban like this:

sudo fail2ban-client status synflood sudo nft list table inet f2b-table

You should see the attacker’s IP in the addr-set-synflood (or similar) set.

The reason this triggered is that ab -n 500 -c 20 (without -k) opens one new TCP connection per request. On a fast link, that translated to thousands of SYNs/sec, which trips a 30 SYN/s hashlimit very quickly.

The Keep-alive Difference (Important)

I reran the same test but with keep-alive enabled:

sudo ab -k -n 500 -c 20 http://victim-ipv4/

This test did not trigger blocking.

With -k, ab reuses the 20 connections (you’ll see Keep-Alive requests: 500 in the output), so only about 20 SYNs are sent rather than 500.

Browsers reuse connections the same way, so this mode is a better approximation of real-world traffic.

Testing `CONNLIMIT` (concurrent connections)

To hit CONNLIMIT you must keep many simultaneous connections open.

My first attempt was:

sudo ab -k -n 2000 -c 101 http://victim-ipv4/

That didn’t hit CONNLIMIT because SYN_FLOOD_PROTECTION blocked the test first — opening 101 connections quickly trips the SYN limit rules.

To test CONNLIMIT properly I temporarily bypassed SYN_FLOOD_PROTECTION for my attacker IP and ran the keep-alive test again:

sudo iptables -I SYN_FLOOD_PROTECTION 1 -s attacker-ipv4 -p tcp -m multiport --dports 80,443 -j ACCEPT sudo ab -k -n 2000 -c 101 http://victim-ipv4/

With that bypass in place, CONNLIMIT logged hits and began dropping connections above 100, which confirmed the connection-cap behavior works as designed.

At the end, I used the sudo ufw reload command to remove the bypass.

IPv4 vs IPv6 behavior

Bans are per address family.

When the attacker IPv4 address was banned, the victim’s IPv6 address still responded normally.

For example, after banning IPv4 I got the default Nginx page when curling the victim’s IPv6:

curl http://[victim-ipv6]/ # -> returns the Nginx index.html content

But curling the victim’s IPv4 hung with no response:

curl http://victim-ipv4/ # -> hangs / no output

If your web server serves both families, you can test both.

Conclusion and Final Thoughts

If you’ve followed everything up to this point – congratulations.

You’ve built a layered firewall that’s far more capable than a stock UFW setup.

Your server now:

Filters malformed or spoofed packets right at the PREROUTING stage.
Whitelists Cloudflare edge IPs for clean traffic prioritization.
Enforces per-IP connection and SYN limits that adapt to activity, not static thresholds.
Reacts to repeated abuse through Fail2ban , which adds offenders to nftables sets instantly.
Escalates bans for repeat offenders (short initial bans that grow with repeat offenses via the recidive logic).
Adds an extra layer of resilience with kernel hardening , so even low-level packet tricks don’t slip through.

All of these layers work together silently.

In testing, the difference is immediate. You can hammer the server with raw SYN floods, or try slow connection hoarding, and still see the web server stay responsive.

At the same time, real visitors and search bots browse normally without ever noticing what’s happening behind the scenes.

I’ll continue extending UFW+ to cover more ports and add UDP/QUIC protections (port 443/UDP) in future versions.

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

Thanks for following along – and if you’ve built your own variation of this setup or tested it in your own environment, I’d love to hear how it performed.

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance.

And if you found this useful, or have thoughts and feedback, drop them in the discussion section – I always enjoy seeing how others build on this.

Until then, keep your packets clean and your logs quiet. See you in the terminal.

How to Use IP Pools for Better Deliverability in Postal

Thu, 05 Jun 2025 00:00:00 GMT

In my Postal SMTP server setup guide, we built a fully functioning send-only mail server and successfully sent our first email using the server’s primary IP address.

Now it’s time to take things further.

In this guide, we’ll learn how to configure IP pools in Postal to send emails from multiple IP addresses instead of relying solely on your server’s main IP.

This setup helps improve deliverability, isolate sender reputation, and add flexibility – especially when working with multiple clients or high-volume sending.

To make this work, we’ll use Floating IPs from Hetzner, which allow you to attach extra IP addresses to your server without losing them if the server is ever rebuilt.

❗

If you’re using a different server provider, make sure they support similar functionality.

We’ll cover everything from assigning and configuring Floating IPs to setting up DNS records, building your IP pools in Postal, and fine-tuning priorities for smart sending behavior.

I’m In!

Why Use IP Pools and Floating IPs?

Using IP pools in Postal along with Floating IPs gives you greater control over how your emails are sent – and boosts deliverability , flexibility , and reputation management.

Here’s why IP pools and Floating IPs are worth setting up:

Combined, they let you build a more robust and professional email sending setup.

Setting Up Floating IPs

To send emails from multiple IP addresses in Postal, you’ll need to set up Floating IPs on your server.

This example walks you through creating and configuring two Floating IP sets (IPv4 + IPv6 pairs) using Hetzner , though the process is similar with other providers that support floating or additional IPs.

👉

New to Hetzner? Use my link to get free credits!

Create Floating IPs in Hetzner

In the Hetzner dashboard, go to Floating IPs from the left menu
Click Add Floating IP
Choose the same location as your Postal server
Add two IP pairs :
1. Two IPv4 addresses
2. One IPv6 block (/64) – you’ll create multiple usable IPv6 addresses from it (e.g. ::1, ::2)
Give each IP set a clear name to stay organized

💡

You can create as many floating IPs as needed and group them however you like. Just make sure to track which IPv4 is paired with which IPv6.

Assign Floating IPs to Your Server

For each IP address:

Click the three-dot menu next to the IP
Choose Assign
Select the server where Postal is installed

❗

This links the IPs to your server at the network level, but they’re not active yet.

Configure the IPs on Your Server

Although the IPs are now assigned, your server still doesn't know how to use them.

You need to manually configure the IPs in your network settings to make them active and routable.

To do this, access the server and create a configuration file inside the /etc/netplan/ directory. You can do this by running:

sudo vim /etc/netplan/60-floating-ips.yaml

Then add your IPs. Replace the addresses below with your actual ones:

network: version: 2 renderer: networkd ethernets: eth0: addresses: \- 89.107.50.13/32 # Floating IPv4 - ip-set-1 \- 2a04:2f8:1c0c:a0a2::1/64 # Floating IPv6 - ip-set-1 \- 90.92.38.139/32 # Floating IPv4 - ip-set-2 \- 2a04:2f8:1c0c:a0a2::2/64 # Floating IPv6 - ip-set-2

Use /32 for IPv4 and /64 for IPv6.

You can create multiple IPv6 addresses from your block by simply appending different values (like ::3, ::4, etc.).

Apply and Verify

Set the proper permissions to avoid receiving a warning:

sudo chmod 600 /etc/netplan/60-floating-ips.yaml

Apply the configuration:

sudo netplan apply

Check that your new IPs are active:

ip addr show

You should see all assigned IPs listed under your network interface (eth0).

🧠

You can repeat this process to add more floating IPs in the future. Just extend the same Netplan file (or use separate ones) and follow the same pattern.

Creating an IP Pool

Before you can use multiple IPs for sending, you need to enable IP pool support in Postal.

Open the Postal configuration file:

sudo vim /opt/postal/config/postal.yml

Under the postal: section, add the following line:

postal: use_ip_pools: true

Save and close the file, then restart Postal:

sudo postal restart

Once Postal is back up, head to the web interface:

From the top menu, click on IP Pools
Click Create the first IP pool
Give your pool a descriptive name (e.g. dedi, client-a, transactional) and click Create IP pool

You’ll now see an empty table where you can start adding IP addresses to this pool.

Adding Two IP Pairs to the Pool

Now that you've configured two Floating IP pairs on your server, it’s time to add them to your IP pool.

If you name your pool dedi** like me, here’s how you can add each IP pair to it:

IP Pair	IPv4	IPv6	Hostname
First IP Pair	`89.107.50.13`	`2a04:2f8:1c0c:a0a2::1`	`dedi-1.example.com`
Second IP Pair	`90.92.38.139`	`2a04:2f8:1c0c:a0a2::2`	`dedi-2.example.com`

To add them:

Click Add an IP address to pool
Enter the IPv4, IPv6, and Hostname for the first pair
Enter a priority value
Click Create IP address
Repeat for the second pair

If you want to use just one specific Floating IP set for sending, make sure only that set is in the IP pool. If you add multiple IP sets to the same pool, Postal will use them all – based on their priority values.

Each IP set in a pool can have a priority between 1 and 100. This priority determines the likelihood that a specific IP will be chosen when sending an email. The higher the number, the more likely it is to be used.

By default, the priority is set to 100.

✅

Once your pool is set up and IPs are added, you're ready to move on to DNS setup.

DNS Setup

Now that you've added your IP pairs to the pool, it's time to configure the DNS records for each one.

This step is crucial – without proper DNS records, your emails might never make it to inboxes or could be flagged as spam.

A and AAAA Records

For each IP pair, you’ll need to create both A and AAAA records that point to the respective IPv4 and IPv6 addresses.

Here’s how you would configure them based on the example IPs and hostnames:

Record Type	Host	Value (IP Address)
A	`dedi-1.example.com`	`89.107.50.13`
AAAA	`dedi-1.example.com`	`2a04:2f8:1c0c:a0a2::1`
A	`dedi-2.example.com`	`90.92.38.139`
AAAA	`dedi-2.example.com`	`2a04:2f8:1c0c:a0a2::1`

Each hostname must resolve to the exact IPs you've assigned in the pool.

PTR (Reverse DNS)

Once you've configured your A and AAAA records, the next step is to set the PTR (Reverse DNS) records for your Floating IPs.

This is important because many mail servers check if your IP address points back to the same domain name you're sending from. If it doesn't match, your email might get flagged as spam or blocked entirely.

Here’s how to do it in Hetzner:

Log in to the Hetzner Cloud Console and navigate to your project.
From the left-hand menu, click Floating IPs.
You’ll see your assigned IPv4 and IPv6 addresses listed.
For each IP:
1. Click the three-dot menu on the right.
2. Choose Edit Reverse DNS.
3. Enter the matching hostname for that IP.

Make sure your PTR record matches the A/AAAA record exactly. This reverse DNS match is a key requirement for being accepted by many recipient mail servers.

💡

For IPv6 addresses , append a 1, 2, etc., after the :: (based on the IP pair).

SPF Record

To improve your email deliverability and avoid getting flagged as spam, you need to configure SPF (Sender Policy Framework) records in your DNS.

SPF helps receiving mail servers verify that your server is authorized to send emails for your domain.

If you followed my Postal setup guide, you should already have a global SPF record for your server’s primary IPs that looks something like this:

Record Type	Host	Value
TXT	`spf.mail.example.com`	`"v=spf1 ip4:70.130.219.212 ip6:643c:ac1f:3f7c:bb5c::1 ~all"`

Now that we’ve created a dedicated IP pool using Floating IPs, we’ll need a separate SPF record for them:

Record Type	Host	Value
TXT	`spf.dedi.example.com`	`"v=spf1 ip4:89.107.50.13 ip6:2a04:2f8:1c0c:a0a2::1 ip4:90.92.38.139 ip6:2a04:2f8:1c0c:a0a2::2 ~all"`

Finally, update your global SPF record to include the new one:

Record Type	Host	Value
TXT	`spf.mail.example.com`	`"v=spf1 ip4:70.130.219.212 ip6:643c:ac1f:3f7c:bb5c::1 include:spf.dedi.example.com ~all"`

This setup ensures that all IPs in your dedi pool are authorized to send emails for any domains you add in Postal.

Why SPF Can’t Fully Isolate IP Pools

If you include both your dedi pool IPs and your primary server IPs in the same SPF, receiving mail servers will treat all of those IPs as valid senders for @example.com – even if, at the Postal level, you’ve told Postal to use only the dedi pool.

In other words, including both in one SPF means you cannot restrict the domain to just the dedi pool.

This means:

No True Isolation: Even if you plan to send some mail only from the dedi pool, the SPF record still authorizes your primary server IPs (and later, any other pools you include). As a result, you cannot stop a hard-coded domain from accidentally using the wrong IP if Postal ever sends from it.
The Return-Path Problem: Postal always uses a single return-path (MAIL FROM) hostname – something like rp.mail.example.com – for every email it sends. The receiving server checks SPF against that return-path hostname. If that hostname’s SPF record includes both your primary and dedi IPs, there’s no way to tell Postal "when sending from the dedi pool, use a different return-path". Postal simply doesn’t let you override the return-path per domain or per pool. As a result, even if you add a separate spf.dedi.example.com record, Postal will still send everything as @rp.mail.example.com, so the only SPF that actually matters is the one on rp.mail.example.com, and it will include all your IPs.

You have two choices:

Live With It: Accept that your SPF lists both (or all) of your IPs under the single return-path hostname. Any sending domain pointing at that SPF can egress on any of them. You get simplicity and fewer records, but you lose the guarantee that Domain A will only ever use the dedi pool.
Run One Postal Instance Per Pool: If you truly need Domain A to use only the dedi IPs and Domain B to use another pool IPs, you must run two separate Postal servers – each with its own hostname and its own rp.* return-path. That way, Postal’s built-in return-path SPF is tied to just one pool, and you regain full separation at the cost of extra infrastructure.

If you’re not running Postal in a mission-critical environment where strict IP separation is absolutely required, it’s perfectly acceptable to live with a combined SPF record.

In practice, the pool-level routing that Postal provides has been reliable, and I’ve never experienced misrouted or mis-authorized mail because of this setup.

If you ever find you need airtight isolation, you can think about the two-instance approach – but for most use cases, trusting Postal’s pool assignment and a single SPF record is simple and works just fine.

Put Your IP Pool Into Action

Now that your IP pool is set up and your DNS is configured, it’s time to put it to use.

Go back to your organization in Postal, and you’ll see a new tab called IPs. Open it and select the IP pool you just created (e.g. dedi). This enables your organization to use that pool for sending across all mail servers associated with it.

If you assign multiple IP pools , you can choose which one to use for each mail server from the Server Settings.

👉

Once you assign an IP pool to an organization, any new mail server you create will no longer use your server’s primary IPs. All sending will happen through the IPs in the assigned pool.

Conclusion and Final Thoughts

That’s it – you’ve successfully set up and configured IP pools in Postal using Floating IPs!

This setup gives you more control over your sending infrastructure, helps protect your IP reputation, and improves deliverability by isolating different types of email traffic.

If you ever need to add more IPs or create additional IP pools (for example, for new clients or projects), just repeat the same process:

Add new Floating IPs
Configure them on your server
Update DNS records (A, AAAA, PTR, SPF)
Add them to an IP pool in Postal
Assign the pool to the appropriate organization

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Installing Docker and Docker Compose on Ubuntu (Using Docker’s Official Repository)

Thu, 01 May 2025 00:00:00 GMT

When I first started with Docker, I used to just run apt install docker.io and call it a day. It worked – but not always well.

The Docker and Compose versions from Ubuntu's default repositories were often outdated , and I quickly ran into missing features and annoying compatibility issues.

One example: the newer Docker Compose v2 plugin (which integrates with the docker command) wasn’t available at all – Ubuntu’s repo still had the old, now-deprecated docker-compose v1 binary.

After a bit of digging, I found the better way: installing Docker Engine and the Compose plugin straight from Docker’s official repository. It’s what the Docker team recommends, and it ensures you're always running the latest stable versions.

In this guide, I’ll walk you through the steps I now use – a clean setup that keeps things up-to-date and future-proof.

I’m In!

Uninstall Old Packages

Before installing Docker from the official repository, it’s a good idea to remove any old or conflicting packages that might be lingering from a previous setup.

You can uninstall them all in one go with this loop:

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt remove -y $pkg done

This command iterates through common legacy Docker and container packages – such as docker.io, docker-compose, containerd, and more – and uninstalls each one.

🙆‍♂️

Don’t worry: This won’t delete your Docker images or containers. It just clears out the old binaries so you're starting fresh.

Once that’s done, clean up any leftover dependencies:

sudo apt autoremove -y

This step ensures any libraries or packages that were only needed by the old Docker binaries are removed, leaving you with a clean slate for the new installation.

Install Docker from Official Docker Repository

Now it’s time to install Docker Engine and the Docker Compose plugin using Docker’s official APT repository – not the outdated packages from Ubuntu’s default sources.

This ensures you get the latest version of Docker CE (Community Edition) along with the integrated Compose v2 plugin.

Run the following commands to add Docker’s GPG key and repository:

# Update your package list and install tools needed for the setup sudo apt update sudo apt install -y ca-certificates curl gnupg # Create a keyring directory (best practice for modern APT key management) sudo install -m 0755 -d /etc/apt/keyrings # Download Docker’s official GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ | sudo tee /etc/apt/keyrings/docker.asc > /dev/null # Set correct permissions sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the Docker repository to APT sources echo \ "deb [arch=$(dpkg --print-architecture) \ signed-by=/etc/apt/keyrings/docker.asc] \ https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" \ | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Refresh package list again sudo apt update

Let’s break down what this does:

Installs required packages like curl and gnupg (if not already present) for handling HTTPS and GPG keys.
Downloads Docker’s official GPG key and saves it in /etc/apt/keyrings/docker.asc (a secure location for APT keys).
Adds Docker’s APT repository to your server’s software sources (pointing to your Ubuntu release codename).
Updates the package list to include Docker’s repository.

Once the repo is set up, install Docker and its core components:

sudo apt install -y docker-ce docker-ce-cli containerd.io \ docker-buildx-plugin docker-compose-plugin

This installs the Docker daemon (docker-ce) along with:

docker-ce-cli: The command-line interface for Docker.
containerd.io: The container runtime Docker uses under the hood.
docker-buildx-plugin: Docker’s Buildx plugin (for extended build capabilities).
docker-compose-plugin: The Docker Compose v2 plugin, which adds the docker compose subcommand.

✅

With this, Docker Engine is installed as a service on your server.

Sometimes Docker doesn’t start automatically after installation. To ensure it’s running, start the service and check its status:

sudo systemctl start docker.service sudo systemctl status docker.service

If the service is listed as active (running), Docker is ready to use – otherwise, you can inspect its logs with sudo journalctl -u docker.service --no-pager to diagnose any startup issues.

Run Docker Hello-World

To verify that Docker Engine is installed and working properly, run the classic hello-world container:

sudo docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints a Hello from Docker! confirmation message and then exits.

If you see this message, congrats – your Docker installation is successful.

Manage Docker as a non-root user

By default, you need to use sudo for all Docker commands. If you’d prefer to run Docker as your normal user (without sudo):

sudo usermod -aG docker $USER

After running this, log out and SSH back in (or reboot) for the group change to take effect.

Once you’ve SSHed in again, test it by running docker info or docker run hello-world again – this time without the sudo prefix.

Verify Docker Compose Plugin

Now that Docker and the Compose plugin are installed, let’s confirm that Docker Compose v2 is working properly.

Run the following command to see if Docker Compose is recognized:

sudo docker compose version

If everything was set up correctly, this will display the Compose plugin’s version, for example:

Docker Compose version v2.x.x

The important part is that you see a version starting with v2, confirming that you have Docker Compose V2 installed as a plugin.

If you see an error:

Double-check that you installed the docker-compose-plugin package.
Make sure you're running the command as sudo or that your user is part of the docker group.

Now, rerun the previous command and verify whether Docker Compose is recognized.

Run a Quick Docker Compose Test

Let’s test Docker Compose by launching a simple container.

Create a test directory:

mkdir docker-compose-test && cd docker-compose-test

Create a docker-compose.yml file with the following content:

services: hello: image: hello-world

This defines a Compose app with one service named hello, using Docker’s tiny hello-world image.

Run the application:

sudo docker compose up

Docker (via the Compose plugin) will pull the hello-world image (if not already pulled) and start the container.

You should see output indicating that Docker is creating and running the container, and then the Hello from Docker!** message from the hello-world container.

Compose will manage the container’s lifecycle. Since hello-world exits after printing its message, you’ll likely see Compose stop the container once it’s done.

For example, the output may look like:

[+] Running 2/2 ✔ Network test_default Created 0.1s ✔ Container test-hello-1 Created 0.1s Attaching to hello-1 hello-1 | hello-1 | Hello from Docker! hello-1 | This message shows that your installation appears to be working correctly. ... hello-1 exited with code 0

Success! This confirms that docker compose is working, and the Compose plugin is running correctly.

In a real-world app, your containers would typically stay running – like a web service using NGINX or a database – but for testing, hello-world is a simple and safe way to verify the setup.

Conclusion and Final Thoughts

By installing Docker from Docker’s official APT repository , you’ve set yourself up with:

The latest Docker Engine
The official Docker Compose v2 plugin
A setup that follows current best practices for Ubuntu servers

This approach not only gives you access to the newest features (like docker compose instead of the old docker-compose binary), but also ensures smoother upgrades moving forward.

💬 Found this guide helpful?

I’d love to hear your thoughts, questions, or suggestions in the discussion section below. Your feedback helps improve future guides and supports others on their server journey.

Prefer a more direct conversation? Feel free to contact me anytime.

Self-Host Private and Lightweight Analytics with Umami

Thu, 17 Apr 2025 00:00:00 GMT

I've always looked for a web analytics solution that respects user privacy and doesn’t slow down my website. The default option for most people is Google Analytics – it’s popular, but in my opinion, it’s bloated and far from privacy-friendly.

Many analytics platforms aren’t GDPR-compliant and often require cookie consent banners, which can hurt the user experience. On top of that, relying on third-party services can negatively affect your site’s performance – just like Google Analytics does.

So why do we keep using third-party tools to collect visitor data when we can host our own analytics and truly own our data?

Sure, some platforms offer deep insights, but unless your business relies heavily on those insights, all that detail isn’t always necessary.

In this guide, I’ll show you how to self-host Umami – an open-source analytics tool that’s fast, privacy-friendly, and easy to host.

Umami No Longer Supports MySQL

As of Umami v3 , MySQL is no longer supported.

In previous versions of this guide, I used MySQL as the database backend.

If you installed Umami following those steps, don’t worry – you can migrate your existing data to PostgreSQL , which is now the only supported database engine.

👇

Follow the new “Migrating from MySQL to PostgreSQL" section to safely switch over before updating to the latest Umami version.

If you’re new to Umami , simply follow this guide from the beginning – it now covers installation with PostgreSQL (the only supported database going forward).

Why Choose Umami?

After using Google Analytics for a couple of months to track my website’s traffic, I just stopped. I said to myself, nah, there has to be a better solution.

It made my website feel heavy and sluggish. I had to add a cookie banner just to stay compliant, and honestly, that hurt the user experience. People would land on my site, see the banner, and bounce – probably never to return.

I’ve always wanted to build a site that feels fast, clean, and respectful. So why should I ruin the experience just to collect some traffic stats?

Eventually, I came across Pirsch.io – a solid, privacy-friendly tool that doesn’t use cookies and doesn’t require consent banners. I used it for a while and liked it.

But if you know me… I’m the self-host guy. I wanted full control – my server, my data, my rules.

That’s when I found Umami. It anonymizes visitor data to protect privacy, doesn’t use cookies (so no annoying cookie banners), and since it’s self-hosted on your own infrastructure, your data stays entirely in your hands.

The best part? Its tracking script is only 2KB in size – practically nothing! Google Analytics used to slow down my site’s page load time by around 500ms.

On top of that , it has a fantastic UI – easy to navigate, simple to understand, and quick to find exactly what you need. It also comes packed with features like reporting, comparison tools, filtering, custom events, team collaboration, and much more.

Requirements Before You Start

To follow this guide, you’ll need a server running Ubuntu 24.04 LTS , prepared for installing Umami. I personally recommend using Hetzner – it's reliable and affordable.

👉

New to Hetzner? Use my link to get free credits!

Make sure you’ve set up DNS records for the domain or subdomain you plan to use for accessing Umami’s web interface:

Add an A record pointing to your server’s IPv4 address.
If you’re using IPv6, also add an AAAA record.

I always recommend running self-hosted projects on the main hostname (like umami.yourdomain.com) if that project is the only thing hosted on the server – which is the case here. I don’t recommend running Umami alongside other services on the same server. Instead, always opt for one server per project when possible. It keeps things clean, reduces conflicts, and makes troubleshooting easier.

You can enable protection for your server’s primary IP addresses in the Hetzner dashboard to ensure they’re preserved even if the server is deleted.

This way, when restoring Umami on a new server, you can reassign the same IPs – avoiding the need to update any DNS settings.

The only major requirement for installing Umami is having Docker Engine and the Docker Compose plugin installed on your server.

👉

For step-by-step instructions, see my Docker Engine & Compose Plugin installation tutorial on Ubuntu Server.

Umami doesn’t come with an automatic installation script – instead, we’ll create a docker-compose.yml file and define the necessary configuration ourselves.

The official repository ships an example PostgreSQL stack, and I use that as a baseline before layering on my own tweaks.

Installing Umami

Installing Umami is a straightforward process.

First, navigate to your home directory and create a new directory for Umami:

mkdir ~/umami cd ~/umami

Next, create a docker-compose.yml file inside that directory and add the following configuration:

services: umami: image: docker.umami.is/umami-software/umami:postgresql-latest ports: \- "3000:3000" environment: DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@umami-postgres:5432/${POSTGRES_DB} DATABASE_TYPE: postgresql APP_SECRET: ${APP_SECRET} depends_on: umami-postgres: condition: service_healthy restart: unless-stopped healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:3000/api/heartbeat || exit 1"] interval: 5s timeout: 5s retries: 5 umami-postgres: image: postgres:15 environment: POSTGRES_DB: ${POSTGRES_DB} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: \- ./umami-postgres-data:/var/lib/postgresql/data restart: unless-stopped healthcheck: test: ["CMD-SHELL", "pg_isready -d ${POSTGRES_DB} -U ${POSTGRES_USER}"] interval: 5s timeout: 5s retries: 5

This docker-compose.yml file defines two services:

umami : the actual Umami app.
umami-postgres : the PostgreSQL database that Umami will use to store analytics data.

I mount the database under ./umami-postgres-data. Feel free to rename it, just stay consistent throughout your environment.

You’ll notice the Postgres image is pinned to postgres:15; that matches Umami’s upstream example compose file and keeps the base install aligned with what the maintainers test against. Once Umami officially bumps their reference stack, you can update this version in lockstep.

The services are connected internally by Docker, and depends_on ensures the database is healthy before Umami starts. All sensitive values like database credentials and secret keys are pulled from environment variables using the ${VARIABLE_NAME} format.

You need to create a .env file in the same directory as your docker-compose.yml with the following variables:

POSTGRES_DB=umami POSTGRES_USER=umami POSTGRES_PASSWORD=userpassword APP_SECRET=yourgeneratedsecretkey REDIS_PASS=redispassword REDIS_URL=redis://:${REDIS_PASS}@redis:6379

That covers both the PostgreSQL credentials and the Redis settings we’ll enable later – feel free to pick stronger values right away so you don’t have to revisit the file.

To generate a secure APP_SECRET value, you can run:

openssl rand -base64 32

Copy the output and paste it as the value for APP_SECRET in your .env file.

Avoid using special characters like !, @, #, or & in your PostgreSQL password, as it can sometimes cause issues when parsed inside Docker or PostgreSQL.

Once your docker-compose.yml and .env files are ready, you can start Umami by running the following command inside the same directory:

sudo docker compose up -d

This will pull the necessary images, start the containers in the background, and get Umami up and running on port 3000.

You can now check if everything is running properly with:

sudo docker ps

This will list all running containers. You should see both the umami and umami-postgres services listed and marked as Up.

Note that the container names may look different from the service names you used in the docker-compose.yml. That’s because the Docker Compose plugin** generates container names using this pattern:

\--

Here’s a quick breakdown:

Part	Value
`project-name`	`umami` (folder name)
`service-name`	`umami`, `umami-postgres`
`index`	`1` (first container)

So if your project folder is named umami, your container names will likely be:

umami-umami-1
umami-umami-postgres-1

If something isn’t working or you want to check what’s going on behind the scenes, use:

sudo docker logs

You can use either the container's name or its ID to check logs for errors or issues.

You can now access Umami’s web interface by visiting:

http://your_server_ip:3000

Username : admin
Password : umami

Before doing anything else, I strongly recommend creating a new admin user with a unique username and strong password – then delete the default one. This helps keep your dashboard secure.

Reverse Proxy Setup

We definitely don’t want to access our web interface using a non-secure connection and our server’s IP address. Instead, we want to use our server’s hostname over a secure HTTPS connection.

To achieve that, we’ll set up a reverse proxy.

You can use any reverse proxy you're comfortable with (like NGINX), but in this guide, we’ll use Caddy. It's a lightweight, modern web server that's easy to configure – and best of all, it handles SSL certificates automatically using Let's Encrypt.

Install Caddy with:

sudo apt install caddy

Caddy’s config is deceptively simple. Start by opening the config file:

sudo vim /etc/caddy/Caddyfile

💡

Make a backup of the file before editing, just in case.

Clear out the default contents and paste in the following config (replace umami.yourdomain.com with your actual domain):

umami.yourdomain.com { reverse_proxy localhost:3000 }

Before restarting Caddy, make sure your firewall actually lets ports 80/443 through; otherwise, Let’s Encrypt can’t validate the certificate:

sudo ufw allow 80/tcp sudo ufw allow 443/tcp

With the rules in place, restart Caddy:

sudo systemctl restart caddy

Caddy will automatically fetch an SSL certificate for your domain and start proxying traffic to your Umami container running on port 3000.

Restrict Direct Access Port 3000

By default, Docker exposes Umami on port 3000, which means it can be accessed directly via your server's IP (http://your_server_ip:3000).

To enhance security and ensure all traffic goes through your reverse proxy (Caddy), you should restrict Umami to listen only on localhost.

In your docker-compose.yml, change the port binding from:

ports: \- "3000:3000"

To:

ports: \- "127.0.0.1:3000:3000"

This ensures Umami is only accessible from inside the server (by Caddy), and blocks external access on port 3000.

Then, from inside your Umami project directory, restart your Docker containers:

sudo docker compose down sudo docker compose up -d

Umami is now securely hidden behind your reverse proxy.

Enabling Redis

Umami supports Redis as a caching layer to improve performance and handle login authentication.

When enabled, Redis caches frequently accessed data like website lookups , reducing database load and speeding up responses. It also replaces JWT tokens with Redis-based session management , providing a more efficient and secure way to handle user sessions.

To enable Redis, we need to edit our docker-compose.yml file to include Redis as a service and connect Umami to it.

First, add a new environment variable under the APP_SECRET variable:

REDIS_URL: ${REDIS_URL}

This tells Umami where to find Redis.

If you followed the earlier .env snippet, you already have the Redis variables in place. Otherwise, add the following lines now (and replace redispassword with something strong):

REDIS_PASS=redispassword REDIS_URL=redis://:${REDIS_PASS}@redis:6379

REDIS_PASS is the single source of truth for both the container and Umami’s connection string, so you only have to set the password once.

Now, add Redis as a service to your docker-compose.yml file. This will run Redis as a container alongside Umami. Place this Redis service definition at the end of the file:

redis: image: redis:latest command: redis-server --requirepass ${REDIS_PASS} volumes: \- ./redis-data:/data restart: unless-stopped healthcheck: test: ["CMD", "redis-cli", "-a", "${REDIS_PASS}", "ping"] interval: 5s timeout: 5s retries: 5

Also, in your Umami service section, add Redis as a dependency to ensure Umami waits for Redis to be ready before starting. Under depends_on, include:

redis: condition: service_healthy

After updating the docker-compose.yml file, apply the changes by restarting your containers:

sudo docker compose down sudo docker compose up -d

Verify that all containers are running:

sudo docker ps

To check that Redis is working and accepting connections, enter the Redis container (substitute the same password you set in REDIS_PASS):

sudo docker exec -it umami-redis-1 redis-cli -a "${REDIS_PASS}"

Inside Redis CLI, test the connection:

ping

If Redis is working, it will respond with:

PONG

Finally, to confirm Umami is connecting to Redis, you can monitor Redis activity:

sudo docker exec -it umami-redis-1 redis-cli -a "${REDIS_PASS}" monitor

Then visit your Umami web interface, and you should see Redis commands appear, confirming that Redis is handling caching and session management.

Installing Updates

To upgrade to the latest version, simply pull the updated image and restart your containers.

Navigate to your Umami project directory and pull the latest image for the PostgreSQL build:

sudo docker pull docker.umami.is/umami-software/umami:postgresql-latest

Recreate your containers using the updated image:

sudo docker compose down sudo docker compose up -d

Umami will apply any necessary database migrations automatically on startup.

If you’re still running the legacy MySQL stack, skip down to the migration section before pulling any new images.

Your data will remain safe, and the update process typically takes just a few seconds.

Always take a snapshot first if you're running in production.

Disaster Recovery

The final step in setting up your self-hosted analytics platform is planning for disaster recovery.

Think about what you’d do if something goes wrong – like a server breach, a misconfiguration, a bad update, or even something out of your control, like a fire in your provider’s data center. You need a way to quickly bring Umami back online without losing your analytics data.

We already have our primary IPs secured (assuming you have enabled protection for them) – they stay with us no matter what, which is great. Now, we need a reliable way to restore the server and reassign those same IPs. The best way to do this is by using snapshots.

If you're using Hetzner, you can go to your server’s Snapshots tab and create one. A snapshot is a full backup of your server’s disk at that point in time.

If disaster strikes, you can spin up a new server from that snapshot, assign the primary IPs, and it should work right away – no need to change any DNS settings or reconfigure Caddy.

Make sure to test this recovery process at least once so you’re confident it works.

Snapshots are created manually. If you want automatic backups, you can enable Hetzner's backup feature. It costs 20% extra, but it keeps daily backups for a week. Once the week is over, old backups are replaced by new ones. You can convert any of these backups into a snapshot, which you can then use to restore your server.

If you delete the server, all its backups are deleted too. Always convert at least one backup into a snapshot before deleting anything. Also, enable deletion protection on your server to avoid losing everything by mistake.

Migrating from MySQL to PostgreSQL

It took me two days to finish this section!

Umami made it unnecessarily hard for Docker users to migrate – just because they didn’t want to spend some time writing a proper walkthrough.

Their official page barely covers the migration and even that doesn’t work out of the box.

What you’ll do (high level):

Spin up a new Umami stack on PostgreSQL using Umami v2.19.0 (the last v2 release).
Import your data from the old MySQL instance.
Verify everything on a temporary port (3001).
Switch the image to v3 and cut over to port 3000.

Why v2.19.0 first? You migrate the data into a clean v2 database, verify, and then upgrade the app to v3. This avoids schema surprises during the import.

First, create a new directory for the PostgreSQL setup:

mkdir umami-postgresql cd umami-postgresql

Inside that directory, create two files: .env and docker-compose.yml.

Open .env and add the following:

POSTGRES_DB=umami POSTGRES_USER=umami POSTGRES_PASSWORD=password APP_SECRET=yourgeneratedsecretkey REDIS_PASS=redispassword REDIS_URL=redis://:${REDIS_PASS}@redis:6379

Open docker-compose.yml and add the following:

services: umami: image: docker.umami.is/umami-software/umami:postgresql-v2.19.0 ports: \- "3001:3000" environment: DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@umami-postgres:5432/${POSTGRES_DB} APP_SECRET: ${APP_SECRET} REDIS_URL: ${REDIS_URL} depends_on: umami-postgres: condition: service_healthy redis: condition: service_healthy restart: unless-stopped healthcheck: test: ["CMD-SHELL", "curl -fsS http://localhost:3000/api/heartbeat || exit 1"] interval: 5s timeout: 5s retries: 5 umami-postgres: image: postgres:15 environment: POSTGRES_DB: ${POSTGRES_DB} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: \- ./umami-pg-data:/var/lib/postgresql/data restart: unless-stopped healthcheck: test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"] interval: 5s timeout: 5s retries: 5 redis: image: redis:latest command: redis-server --requirepass ${REDIS_PASS} volumes: \- ./redis-data:/data restart: unless-stopped healthcheck: test: ["CMD", "redis-cli", "-a", "${REDIS_PASS}", "ping"] interval: 5s timeout: 5s retries: 5

We’re spinning up a new Umami instance running on port 3001 so we can check the migration before replacing the old one.

This setup uses the last MySQL-compatible Umami version (v2.19.0).

Run the containers:

sudo docker compose up -d

Then open your browser and go to:

http://yourserverip:3001

If it loads, you’re good.

Now stop both Umami (app) containers (not PostgreSQL or MySQL or Redis):

sudo docker ps sudo docker stop

From the old MySQL docker directory, run:

docker exec -i \ mysqldump -h 127.0.0.1 -u root -p'YOUR_OLD_ROOT_PASSWORD' \ \--no-create-info --default-character-set=utf8mb4 --quick --single-transaction --skip-add-locks \ umami > umami_mysql_dump.sql

Make the dump Postgres-friendly:

# 1) MySQL backticks -> Postgres double quotes sed -i 's/`/"/g' umami_mysql_dump.sql # 2) MySQL-style escaped quotes -> Postgres-style sed -i "s/\\\\\\\'/''/g" umami_mysql_dump.sql

Copy the dump file into the new Postgres container:

# back in the new umami-postgresql directory sudo docker cp umami_mysql_dump.sql :/tmp/dump.sql

Clear two internal tables (per Umami’s guidance):

sudo docker exec -it \ psql -U -d umami \ -c 'TRUNCATE TABLE "_prisma_migrations", "user";'

Import (temporarily relaxing foreign keys during the load):

```bash sudo docker start ```

Open `http://yourserverip:3001` and log in with your **existing**  Umami credentials from MySQL. If websites don’t appear, make sure you’re logged in as the user who owns them.

Now, edit your `docker-compose.yml` to use the latest image and restore the local binding:

```yaml
image: docker.umami.is/umami-software/umami:postgresql-latest ports: \- "127.0.0.1:3000:3000"

Back in your old Umami directory:

sudo docker compose down -v --rmi local --remove-orphans sudo rm -rf ./umami-db-data ./redis-data

Those folder names match the original MySQL-era guide (umami-db-data). If your old stack used a different directory – say, ./umami-mysql-data – swap the paths accordingly so you don’t delete the wrong data.

Then, back in your new Umami directory:

sudo docker compose up -d

Your Umami instance is now fully running on PostgreSQL – finally, the way it should’ve been documented in the first place.

Conclusion and Final Thoughts

With your Umami instance now fully deployed and secured behind a reverse proxy, you have complete control over your analytics.

From here, you can now start adding websites, creating new teams, setting up custom events, and exploring your analytics dashboard – all from your own infrastructure, fully in your control.

You’ve built a powerful, privacy-friendly analytics platform – no third-party trackers, no compromises.

And just as important – test your disaster recovery plan regularly to make sure everything still works as expected. It’s better to catch issues during a test than during a real outage.

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Setting Up Postal as an SMTP Server

Mon, 07 Apr 2025 00:00:00 GMT

Over the past few weeks, I’ve been exploring the idea of running my own mail server. While I had a basic understanding of how email servers work, I’d never actually set one up from scratch.

To keep things simple (and sane), I decided to focus on just one piece of the puzzle: sending email. No inboxes, no IMAP or POP – just a streamlined, send-only SMTP server.

Get updates from my mail server journey – tips, lessons, and discoveries along the way.

I’m In!

Why Postal?

If you know me, you know that from my collection of server security guides, I’ve always tried to limit the use of third-party security software and prefer to build my own solutions.

I’ve always leaned towards the old school way of doing things, avoiding extra software and instead using the available server packages to suit my needs.

So why didn’t I just go with Postfix for this SMTP setup? I seriously considered it – but Postal offered too many advantages to ignore.

Unlike Postfix, Postal comes with a suite of modern features out of the box. And since this is a fairly complex project, I didn’t want to spend hours wiring together manual configs for every little piece.

Postal simplifies a lot of that – and has an active community (GitHub Discussions) where you can get support if you hit a wall.

🙆‍♂️

One feature that really sold me: IP pool support.

Postal makes it easy to create multiple IP pools, so you can send emails from different IP addresses instead of being stuck with just the server’s main IP.

This is huge for:

Separating email traffic across clients or apps.
Managing deliverability and sender reputation.
Assigning clean IPs to new users without affecting your existing ones.

If you're running multiple projects or managing email for clients, this kind of flexibility is a must-have.

👉

And that’s just scratching the surface.

Throughout this guide, you’ll discover more powerful features , such as its clean and intuitive web interface, its built-in integration with SpamAssassin to scan outgoing emails for spam, and even the ability to receive emails through routes (yep, we’ll cover that too).

Author's Note

Now that you know why I chose Postal as my SMTP server, I want to highlight a few key points that can make or break your setup – especially when it comes to deliverability.

❗

Your IP reputation is everything.

A clean, trusted IP is crucial – not just for your main server, but for every IP you use in a pool. A poor reputation can lead to your emails being flagged as spam or blocked entirely.

That’s why choosing the right server provider matters. In my experience, Hetzner has the strictest email-sending policies.

Most providers block port 25 by default to combat spam, and while many will unblock it upon request, Hetzner is by far the most stringent in this regard. Their strict policies ensure that their IPs remain clean and trusted, making them an excellent choice for an SMTP server.

👉

New to Hetzner? Use my link to get free credits!

But even with clean IPs, don’t expect inbox success on day one. If you're using a brand-new IP, mail servers may treat it as suspicious. You’ll need to warm up the IP – sending gradually and building trust over time.

🔁

Are you planning to use multiple IPs or create IP pools? Choose a provider that supports it.

Hetzner, again, is great here. They offer Floating IPs –** extra public IP addresses you can attach to your VPS in addition to the main IP.

What makes Floating IPs useful:

They’re detached from the server itself, so if you delete or replace the server, you can reassign the same IPs to a new one.
This helps you preserve your sending reputation and avoid restarting from scratch.
Perfect for IP pools and advanced sending setups.

If you're using a provider other than Hetzner, be sure to check if they offer something similar.

✅

And don’t forget: Make sure port 25 is open. Without it, your SMTP server won’t be able to send anything.

IP Pools

If you're interested in using IP pools with Postal, I've written a separate guide that walks through how to set them up and use them effectively.

But don’t worry about that just yet – I recommend completing this setup guide first , then coming back to the IP pools once everything is up and running.

👉

How to Use IP Pools for Better Deliverability in Postal

Server Preparation

Before preparing our server to run Postal, we need to deploy it first.

Choose Ubuntu 24.04 LTS as the server image – it’s stable, well-supported, and perfect for this setup.

If you're using Hetzner, you’ll notice they offer two types of servers:

Shared CPU (resources are shared with other customers)
Dedicated CPU (resources are reserved for you)

❗

You don’t want your outbound queue throttled because of a noisy neighbor.

I highly recommend opting for the servers with dedicated CPUs , as an SMTP server is mission-critical and should never face resource limitations.

It’s also essential to enable IPv6 on your server (many modern mail servers require it) and monitor server resources to avoid bottlenecks in production.

Initial Setup

After your server is deployed, SSH into it and begin by running the following commands to perform a full update:

apt update apt dist-upgrade

💡

Assuming you’re accessing the server as root for the first time, remember to change your root password to something stronger afterward.

Next, make sure to set the correct timezone for your server. This isn’t just for convenience – it’s important for Postal’s internal logging and email timestamps.

For example, to set the timezone to Berlin, you can run:

timedatectl set-timezone Europe/Berlin

With the timezone in place, the next step is to configure your server’s hostname.

A typical hostname consists of two parts: the server name and the domain name. Since we’re building an SMTP server, a clear and descriptive hostname helps other mail servers recognize your server’s role.

A recommended format is:

hostnamectl set-hostname mailout.example.com

Using mailout as the server name is a good practice, as it clearly indicates to other mail servers that this server is handling SMTP functions.

For better security:

Create a new user with sudo privileges
Disable the root user
Set up SSH keys for the new user
Disable password authentication

👉

I’ve covered all of this in detail in my Server Preparation guide – check it out if you need help.

Install Dependencies and Docker

Now that your server is updated and configured, let’s install the essential tools Postal relies on.

Start by installing the required packages:

sudo apt install git curl jq

This ensures that git is available on your server, which we’ll use to clone the Postal installation helper repository:

sudo git clone https://github.com/postalserver/install /opt/postal/install

Then, create a symbolic link so you can run the postal command from anywhere on the server:

sudo ln -s /opt/postal/install/bin/postal /usr/bin/postal

💡

The postal command needs to be run with sudo privileges or as the root user.

Use a Proper Docker Setup (Not Ubuntu’s Built-in Version)

Postal runs entirely inside containers, so installing Docker Engine and the Docker Compose plugin is a major prerequisite.

But, the Docker packages included in Ubuntu’s default repositories are often outdated and may cause issues.

To avoid problems, follow my Docker installation tutorial, where I walk through installing the latest version of Docker and Docker Compose the right way – fully compatible with Postal.

Set Up MariaDB

Postal requires a database engine to store all emails and other essential configuration data, and it will automatically provision a database for each mail server you create.

We’ll run MariaDB in a container, which is the easiest option:

sudo docker run -d \ \--name postal-mariadb \ -p 127.0.0.1:3306:3306 \ \--restart always \ -e MARIADB_DATABASE=postal \ -e MARIADB_ROOT_PASSWORD=postal \ mariadb

This will start a MariaDB instance, listening on port 3306. Make sure to change the root password to a strong one and store it securely, as you’ll need it later during the Postal configuration.

You can verify that MariaDB is running with:

sudo docker ps

You should see the postal-mariadb container listed with its status as Up.

Final Prep Steps

Once everything is in place, it's a good idea to sudo reboot your server to make sure all changes are fully applied.

If you're using Hetzner , visit the Primary IPs tab and enable protection for your assigned IPv4 and IPv6 addresses.

✅

IP protection allows you to reuse your existing IPs if you ever need to redeploy the server.

Generating Configuration Files

Before starting the installation, Postal provides a tool that automatically generates the initial configuration files required for setup.

Run the following command, replacing mailout.example.com with the actual hostname you set earlier – the one you plan to use to access the Postal web interface:

sudo postal bootstrap mailout.example.com

The command will generate three important files inside the /opt/postal/config/ directory:

postal.yml : The main configuration file for Postal.
signing.key : A private key used by Postal to sign various internal components.
Caddyfile : The configuration file for the Caddy web server, which Postal uses to serve its web interface.

The only file that matters to us at this point is the postal.yml file, which we’ll use to tweak and customize Postal’s configuration.

Once the files are generated, open the postal.yml file and update the passwords for both the main_db and message_db sections.

Use the password you specified earlier when creating the MariaDB instance with Docker:

main_db: host: 127.0.0.1 username: root password: here_comes_the_password database: postal message_db: host: 127.0.0.1 username: root password: here_comes_the_password prefix: postal

Save and close the file – that’s all we need to do for now.

❗

Since the Docker setup mounts /opt/postal/config as /config, any file paths referenced inside postal.yml should start with /config instead of /opt/postal/config.

DNS Setup

Correct DNS configuration is critical. If your DNS isn’t properly set up, your emails may end up in spam – or not be delivered at all.

Postal requires several DNS records to function properly. Let’s walk through each type.

A & AAAA

We’ll begin by creating two essential DNS records: an A record and an AAAA record:

Record Type	Host	Value (IP Address)
A	mailout.example.com	Primary IPv4
AAAA	mailout.example.com	Primary IPv6

Then, add A and AAAA records for the MX hostname that we will use later for receiving emails through routes:

Record Type	Host	Value (IP Address)
A	mx.mailout.example.com	Primary IPv4
AAAA	mx.mailout.example.com	Primary IPv6

All records should point to your server’s primary IPs

PTR (Reverse DNS)

PTR (Reverse DNS) records are used by receiving mail servers to verify that your server’s IP address maps back to its hostname – this is a key check for spam prevention and essential for passing many email reputation filters.

Now, set up reverse DNS (PTR) for both IPv4 and IPv6:

Go to Hetzner dashboard →Server Details →Networking tab
Find the IP you want to edit, click the three dots →Edit Reverse DNS
Set the reverse DNS to match your hostname (mailout.example.com)
For IPv6, append ::1 and update the hostname accordingly

✅

Your PTR record must match your server's hostname exactly.

SPF Record

It's crucial to add an SPF (Sender Policy Framework) record to your DNS setup.

This record helps prevent your emails from being marked as spam by verifying that your server is authorized to send emails on behalf of your domain.

Add a global SPF record that includes your primary IPs:

Record Type	Host	Value
TXT	spf.mailout.example.com	"v=spf1 ip4:70.130.219.212 ip6:643c:ac1f:3f7c:bb5c::1 ~all"

This is a global SPF record you will use for any domain you add to Postal.

Return Path (MX + SPF + DKIM)

The Return Path is the email address that gets used to handle bounces (rejected or undelivered emails) for any email you send.

It tells the receiving mail server where to send error messages if it can't deliver an email (like if the recipient's email address doesn't exist, or the mailbox is full).

It's different from the From address and is usually hidden from end users – but it’s essential for proper bounce handling.

Think of it like this:

You send an email to someone.
If the email can't be delivered, a bounce email will be sent back to the return path address, not your From address.

To configure a return path domain in Postal, we need to add three DNS records:

Record Type	Host	Mail Server	Priority
MX	rp.mailout.example.com	mailout.example.com	10

This tells the world where to send bounce emails for the return path.

It means bounces for emails sent from mailout.example.com will go to rp.mailout.example.com (your return path domain), which will then point to your Postal server to handle them

Record Type	Host	Value
TXT	rp.mailout.example.com	"v=spf1 a mx include:spf.mailout.example.com ~all"

This authorizes your return path domain to send emails and helps prevent bounce messages from being flagged as spam.

The last DNS record we need to add is the DKIM (DomainKeys Identified Mail) record. It is an email authentication method that uses a digital signature to verify that the email was sent and authorized by the domain owner.

✅

This digitally signs emails from the return path domain, proving they haven’t been tampered with and are authorized by your server.

Postal provides a command that generates a DKIM record for us, which begins with the postal DKIM identifier. If you'd like to customize this identifier, open the /opt/postal/config/postal.yml file and add the following to the dns: section:

dns: dkim_identifier: custom

Afterward, run this command to generate the DKIM record:

sudo postal default-dkim-record

In my case, I am adding a TXT record with the host value of:

custom._domainkey.rp.mailout.example.com

The value for this record will be the output generated by the command.

Route Domain

If you wish to receive incoming emails by forwarding them directly to routes in Postal, you'll need to configure a domain and point it to your server using an MX record, like this:

Record Type	Host	Mail Server	Priority
MX	routes.mailout.example.com	mailout.example.com	10

This lets incoming emails sent to routes.mailout.example.com reach Postal for routing.

DMARC

DMARC (Domain-based Message Authentication, Reporting, and Conformance) is an email authentication protocol that helps prevent email spoofing and phishing by allowing domain owners to specify how incoming emails should be handled if it fails SPF or DKIM checks.

The DMARC record specifies:

Whether to accept, quarantine, or reject emails that fail authentication.
How the domain owner wants to receive reports about these emails.

For this, add a TXT record with a host value starting with _dmarc for your domain and containing the following:

"v=DMARC1; p=quarantine; rua=mailto:postmaster@example.com"

p=quarantine : This policy tells receiving mail servers to treat emails that fail DMARC checks as suspicious and place them in the recipient's spam or junk folder.
rua=mailto:postmaster@example.com : This specifies the email address where aggregate reports about DMARC failures will be sent, allowing the domain owner to monitor and analyze the performance of their DMARC policy.

Adding a DMARC record improves both your domain's security and email deliverability.

❗

Postal doesn't auto-suggest a DMARC record – make sure to add this manually per domain.

Verify DNS Configuration

If you open the /opt/postal/config/postal.yml file, you will see DNS records specified under the dns: section.

This section essentially tells Postal which DNS records you’ve added and how Postal should recognize them.

By default, it includes all the necessary records – except for the Track Domain , which is used for click and open tracking. Since that’s outside the scope of this guide, you can safely comment it out.

✅

Make sure the listed records match your DNS settings. Once they do, your DNS setup is complete.

Postal also supports a couple of optional DNS-related settings you can customize in the same file:

dns: domain_verify_prefix: custom-verification custom_return_path_prefix: customrp

By default, domain_verify_prefix is set to postal-verification and custom_return_path_prefix to psrp. You can personalize these values to reflect your domain or brand.

Starting Postal

With everything configured, it’s time to bring Postal to life.

First, initialize the database and create your admin user:

sudo postal initialize sudo postal make-user

Once that’s done, start Postal:

sudo postal start

This command launches all the necessary components using Docker containers.

You can verify that everything is running smoothly by checking the service status:

sudo postal status

If you run the sudo docker ps command, you’ll notice several new containers up and running – these are the components that power your Postal installation.

Web Access with Caddy

To access Postal’s web interface, you’ll need to set up a web proxy.

You can use any reverse proxy you're comfortable with, but in this guide, we’ll use Caddy.

It is a lightweight, modern web server that’s easy to configure and automatically handles SSL certificates via Let's Encrypt.

Run the following command to start the Caddy container:

sudo docker run -d \ \--name postal-caddy \ \--restart always \ \--network host \ -v /opt/postal/config/Caddyfile:/etc/caddy/Caddyfile \ -v /opt/postal/caddy-data:/data \ caddy

Once it’s running, Caddy will automatically request and install an SSL certificate for your domain.

You should now be able to access the Postal web interface using your hostname and log in with the admin user you created earlier.

Securing the Web Interface

Someone asked about having a kind of 2FA for the web interface, since Postal doesn't provide any 2FA feature for now, and brought up a really good point.

You can restrict access to ports 80 and 443 to a static IP (like a VPN), but this doesn't work for everyone.

So, to add an extra layer of security to the normal username and password authentication method that Postal provides, we can enable Basic Authentication through Caddy, which is easy to configure.

Inside the /opt/postal/config/ directory, you'll find the Caddyfile file. Open it and add the following, so the configuration looks like this:

mailout.examle.com { reverse_proxy 127.0.0.1:5000 basicauth { username $2a$14$dp6WI0ldDiY/lFUL5I7q7ug/29DbwHYe5oFO6Z6mN6SRDIz/1/lgK } }

Replace username with the username you want to use, and what follows the username is the password, but hashed using bcrypt , which is what Caddy uses.

While the caddy hash-password command works normally, it won’t work in this case since we run Caddy in a container, and the command is not available. As a quick alternative, you can use this online tool to hash your password.

After generating the hash, replace the hash in the example above with your own, then save and close the file.

Now, run the following two commands to stop and start Caddy again:

sudo docker stop postal-caddy sudo docker start postal-caddy

Finally, try to access the web interface, and you should not be able to do so unless you enter the username and password you just added to the Caddyfile file.

✅

I hope this helps solve the issue of not having 2FA. Thanks to Martin Kurz for raising this point!

Time to Mail It!

With Postal installed and running, it’s time to test your setup.

Start by creating an organization. Give it a name and click Create Organization.

Once inside:

Go to the Settings tab
Update the timezone to match your local time (this keeps log timestamps accurate)

Simple but important.

A Quick Note on Domains

Before adding your first mail server, there's something important to note.

You can add a domain:

At the organization level : applies to all mail servers in that organization.
Inside a specific mail server : applies only to that one server.

✅

Best practice: Add domains inside each mail server. This keeps things organized and avoids DNS conflicts.

DNS can get complicated and busy when the domain is linked to all mail servers created within the organization.This is something I was advised by a member of the Postal team on GitHub.

If that ever changes and I find a good use case for organization-level domains, I’ll update this guide. But for now, let’s keep it tidy.

Create a Mail Server

Go ahead and create your first mail server. When you do:

Set the Mode to Live
Once it’s created, head to the Domains tab
Add your sending domain (usually the same one you used to set up Postal)

I always add my main domain here and set up a route for postmaster@example.com – that way, I can receive DMARC reports as mentioned earlier in the DNS section.

Inside your mail server, go to Settings → Server Settings , and you’ll find a field called POSTMASTER :

This is the contact address that shows up in bounce messages if someone sends email to a nonexistent address.
It defaults to postmaster@example.com, and that’s exactly what most servers expect – so I recommend keeping it as is.

For example, if someone tries to email an address you haven’t set a route for, it will bounce, and the bounce message will mention the POSTMASTER address as the contact point.

👆

What mentioned above is only relevant if you want Postal to handle incoming emails and forward them to another address, like your Gmail , using routes.

That brings us to the MX record :

If you're not using Postal to receive mail: you can skip adding the MX record Postal suggests when adding a new domain.
If you're not using another service for incoming mail: I still recommend adding the MX record.

It helps make your sending email address appear more trustworthy and legitimate.

Send a Test Email

Let’s make sure it actually works.

Go to the Messages tab → Send Message
In the FROM field, use something like hi@example.com (make sure that domain has been added)
For the TO field, use a spam-checking tool like mail-tester.com
1. It’ll give you a temporary email address
2. Paste that into the TO field and click Send Message

Then head back to the test tool and check your score.

🥳

I got a 10/10. If your score is lower, don’t worry – the tool gives you a breakdown of what went wrong and how to fix it.

Configuring Routes

It's time to set up routes so you can receive incoming mail.

Inside your mail server, go to the Routing tab. You’ll see three endpoint types available:

HTTP
SMTP
Address

For now, we’ll focus on the Address endpoint – it’s the simplest and most practical option for forwarding emails to your inbox.

🧪

Feel free to experiment with HTTP or SMTP endpoints later. If you discover something cool, share it in the discussion section at the end. I may update this guide after deeper testing.

Create an address endpoint with the email address you'd like incoming emails forwarded to – this can be your personal or business inbox.

Create a Route

Now go to the Routes sub-tab and click Add your first route.

Fill in the following fields:

Name : The email address you want to receive emails at (e.g. postmaster). If you've already added your domain, just select it from the dropdown.
Endpoint : Select the address endpoint you just created.
Spam Mode : Choose your preferred option (we’ll configure spam filtering later).
Click Create Route.

💡

postmaster@yourdomain.com is useful for receiving DMARC reports and bounce notifications.

Test Your Route

Try sending an email to the address you configured – e.g. postmaster@example.com and check if it lands in the inbox of the address you configured as the endpoint.

You can also view incoming mail in Postal:

Go to the Messages tab → Incoming Messages

If you open a mail, you’ll first see a summary – like whether it's marked as spam.

A Quick Security Check

Now head to the Outgoing Messages tab and open any mail you’ve sent.

In the Properties tab, you’ll see delivery info – like whether it was sent over a secure (SSL) connection.

But here's the catch: incoming mail don’t show this same detail.

😬

That means your server isn’t receiving mail over a secure connection yet.

We’ll fix that next.

SMTP TLS

By default, Postaldoes not have TLS enabled for incoming mail , which means your server isn’t currently receiving emails securely.

You can confirm this using the CheckTLS tool. Just enter your domain in the eMail Target field and click the Run Test button.

The test should return an error like:

TLS is not an option on this server

You can also use this command to confirm that no SSL is configured and TLS isn’t working:

openssl s_client -connect mailout.example.com:25 -starttls smtp

This command connects to your mail server on port 25 and tries to upgrade the connection to a secure one using TLS.

If TLS isn’t enabled yet, you’ll see an error like:

Didn't find STARTTLS in server response, trying anyway...

This output means your server isn’t advertising TLS as an option, so incoming connections can’t be encrypted – exactly what we’re trying to fix next.

Why This Happens

I spent three days debugging this exact issue, only to realize it was something simple – and poorly documented in Postal’s official docs.

Here’s what you need to know:

When sending email, Postal doesn’t need an SSL certificate – the receiving server handles encryption.
But when receiving email, your Postal server is the receiving server – so you must provide a certificate to support TLS.

💡

That’s why outgoing emails show Received over an SSL connection in the Properties tab, but incoming ones don’t.

When using routes to receive emails, Postal first receives the message, then forwards it to the address endpoint you specified. This forwarding process is encrypted using an SSL certificate, because – as mentioned above – during forwarding, Postal acts as the sending server , and the receiving server (e.g. Gmail) handles the encryption. However, the initial connection between the original sender and Postal is not secure unless SMTP TLS is enabled.

Also important: without TLS, apps like WordPress won’t be able to submit email to Postal securely – so enabling it is strongly recommended, even if you're not receiving mail directly. Because WordPress first submits the email to Postal, which then forwards it to the final recipient.

Issue an SSL Certificate

We need an SSL certificate for both the hostname and the MX record. To obtain one, we’ll use Certbot.

Start by installing it:

sudo apt install certbot

When issuing a certificate, Certbot needs to use port 80 for domain verification. However, that port is currently in use by the Caddy web server, which is handling web traffic for Postal's web interface.

To work around this, we’ll temporarily stop the Caddy container, issue the certificate, and then start the container again – all in one command:

sudo docker stop postal-caddy && sudo certbot certonly --standalone -d mailout.example.com -d mx.mailout.example.com --agree-tos --email postmaster@example.com --non-interactive && sudo docker start postal-caddy

Once that completes, your certificate will be successfully issued.

✅

You can check the certificate and obtain the certificate and key paths along with other relevant details by running the sudo certbot certificates command.

Configure Postal to Use TLS

Now that we have our SSL certificate, we need to copy it to the /opt/postal/config/ directory and rename its certificate and key files for Postal to use:

sudo cp /etc/letsencrypt/live/mailout.example.com/fullchain.pem /opt/postal/config/smtp.crt sudo cp /etc/letsencrypt/live/mailout.example.com/privkey.pem /opt/postal/config/smtp.key

Next, change the file permissions for the certificate files:

sudo chmod 644 /opt/postal/config/smtp.*

Then, open the /opt/postal/config/postal.yml file and add the following:

smtp_server: tls_enabled: true tls_certificate_path: /config/smtp.crt tls_private_key_path: /config/smtp.key

Finally, restart Postal to apply the changes:

sudo postal restart

To verify that SMTP TLS is working, run this command again:

openssl s_client -connect mailout.example.com:25 -starttls smtp

You should now see a successful TLS connection.

Next, run the CheckTLS test again by entering your domain as the email target and clicking Run Test. Check the results to confirm that TLS is properly enabled and functioning.

💡

Try your route again, and now you should see an indicator under the Properties tab that the email was received over an SSL connection.

Handle Auto-Renewal with Certbot

Since Certbot uses port 80 and Caddy is running, the auto-renewal will fail unless you automate stopping/starting Caddy.

If you're curious, the configuration for auto-renewal can be found inside the /etc/letsencrypt/renewal directory.

Certbot handles auto-renewals using systemd timers instead of cron jobs. This timer run twice per day and can be checked using the following command:

sudo systemctl list-timers

You can examine the content of the timer with the following command:

sudo cat /lib/systemd/system/certbot.timer

Now, to solve this problem, we can use something called Renewal Hooks.

Inside the /etc/letsencrypt/renewal-hooks directory, you’ll find three different subdirectories:

deploy : This directory is for scripts that run after a certificate is successfully renewed.
post : Scripts in this directory run after the certificate is renewed and deployed.
pre : These are scripts that run before Certbot tries to renew the certificate.

✅

We use pre to stop the Caddy container, deploy to copy the renewed certificate files, change their permissions, and restart Postal, and post for starting the Caddy container again.

Inside the pre directory, create a script called stop_caddy.sh with the following content:

#!/bin/bash docker stop postal-caddy

Inside the deploy directory, create a script called copy_ssl_cert.sh with the following content:

#!/bin/bash # Copy new certificate files cp /etc/letsencrypt/live/mailout.example.com/fullchain.pem /opt/postal/config/smtp.crt cp /etc/letsencrypt/live/mailout.example.com/privkey.pem /opt/postal/config/smtp.key # Set correct permissions chmod 644 /opt/postal/config/smtp.* # Restart Postal postal restart

Finally, inside the post directory, create a script called start_caddy.sh with the following content:

#!/bin/bash docker start postal-caddy

💡

Don't forget to give each script execute permissions but make sure to restrict them to root only.

You can test if everything is working by first checking the expiry date using the following command:

sudo certbot certificates

This should match the expiry date of the copied certificate. To check the expiry date of the certificate in Postal, run:

openssl x509 -enddate -noout -in /opt/postal/config/smtp.crt

Then, force the renewal using the following command:

sudo certbot renew --force-renewal

Lastly, check the expiry date again to make sure it matches. If it does, the setup is working correctly.

❗

Don’t test this on the same day you issued the SSL certificate. Wait a few days or you won’t see a change in expiry, even if the copy succeeded.

Installing SpamAssassin

For filtering spam emails, we will install and use SpamAssassin , a powerful and widely-used spam filtering tool.

By default, Postal will communicate with SpamAssassin's spamd service using a TCP socket connection (port 783).

You’ll need to install SpamAssassin on your server and enable it within Postal to begin filtering spam.

For installing SpamAssassin, use the following command:

sudo apt install spamassassin

Next, you will need to enable the SpamAssassin timer:

sudo systemctl enable --now spamassassin-maintenance.timer

It is used to update the spam rules automatically.

To enable spam checking, we need to add the following to the end of the /opt/postal/config/postal.yml file:

spamd: enabled: true host: 127.0.0.1 port: 783

Finally, restart Postal to apply your changes:

sudo postal restart

Inside Postal web interface:

Go to your Organization Settings → Spam tab
You’ll see the Spam Threshold setting (default: 5)

A score above the threshold triggers the SPAM MODE set in your route config. A score below the threshold means the message is treated as non-spam.

💡

Remember when you created your first route and selected a SPAM MODE? That setting now takes effect.

There’s also a Spam Failure Threshold (default: 20). If a mail scores above 20, it is immediately dropped.

❗

By default, Postal doesn't filter outgoing emails, only incoming ones, after you install and enable SpamAssassin.

What Do Spam Scores Mean?

Now, let's talk about what spam scores are before we enable spam checking for outgoing emails.

SpamAssassin works by analyzing an email and giving it a spam score.

The lower the score, the better the chances of the email getting delivered successfully.
Conversely, the higher the score, the higher the likelihood the message is labeled spam/junk.
A score below 5 is considered decent, while anything above 5 indicates a higher chance of being filtered out.

SpamAssassin uses over 700 tests and a variety of analytical techniques to detect spam. Each attribute it checks contributes to the overall score.

Negative scores indicate the email is less likely to be spam, while positive scores suggest possible spam.

💡

A score of 0 is neutral, meaning the factor has little impact.

When setting up SpamAssassin, the score threshold can be adjusted, with 5 being the default.

To ensure successful email delivery, aim for a spam score below 5, with scores between 0-2 being optimal. Negative scores are even better, as they indicate the email is very unlikely to be marked as spam, though they can be difficult to achieve.

I recommend starting with a threshold of 5 and gradually lowering it based on your spam checking history. For instance, if you notice your outgoing emails consistently have a score of 3, consider lowering your threshold to 4.

Over time, work on improving your score and address the factors contributing to higher scores.

Enable Spam Checking for Outgoing Mail

To filter outgoing mail:

Go to your mail serverSettings → Advanced Settings
Find the Outbound Spam Threshold
Set it to 5
Click Save Server

Now outgoing mail will also be checked using SpamAssassin before it’s sent out.

✅

This is helpful for catching mistakes early, like misconfigured headers, bad links, or overly promotional content.

System Email Configuration

To enable system-generated emails (such as password reset links) in Postal, you'll need to configure the smtp: section in the Postal configuration file.

Open the /opt/postal/config/postal.yml file and scroll to the bottom of it, and you’ll find the smtp: section. This is where you'll define the SMTP server Postal should use for sending system emails.

Although you could use an external SMTP provider, we’re going to use Postal itself to keep everything self-hosted.

Inside your Postal web interface:

Go to your Mail Server
Click the Credentials tab
Click Add Credential
1. Leave the TYPE set to SMTP
2. Name it something like System to keep things organized
Click Create Credential – a password will be generated automatically

Now go to the Help tab → Sending Email sub-tab. You’ll see the SMTP settings needed to complete the smtp: section in your postal.yml file.

Back in /opt/postal/config/postal.yml, fill out the smtp: section with the info you just retrieved. For the from_address, it's a good idea to use something like system@example.com.

Make sure to also change the port from 2525 to 25, since our SMTP server uses port 25 for sending emails.

Finally, restart Postal to apply your changes:

sudo postal restart

Now, try logging out of the Postal web interface and resetting your password. You should receive an email with a link to set a new one.

Enabling UFW Firewall

We finally want to enable the UFW firewall on our server, and I've left this to the end so you can check if everything still works perfectly after enabling it.

We only need to allow incoming connections, as outgoing traffic is allowed by default with UFW.

To configure this, we need to allow incoming traffic on the following ports: 22 (SSH), 25 (SMTP), 80 (HTTP), and 443 (HTTPS).

Use the following commands to allow these connections and enable the firewall:

sudo ufw limit 22/tcp sudo ufw allow 25/tcp sudo ufw allow 80/tcp sudo ufw allow 443/tcp sudo ufw enable

Sometimes, UFW may require a reboot to work correctly. So, if you encounter any issues, try rebooting the server.

Disaster Recovery

The final step in setting up your SMTP server is planning for disaster recovery. Think about what you’d do if something goes wrong – like a server breach, a misconfiguration, a bad update, or even something out of your control, like a fire in your provider’s data center. You need a way to quickly bring everything back online.

We already have our primary IPs secured – they stay with us no matter what, which is great. Now, we need a reliable way to restore the server and assign those same IPs to it. The best way to do this is by using snapshots.

In Hetzner, you can go to your server’s Snapshots tab and create a snapshot, which is basically a copy of your server's disk at that moment.

If something happens, you can spin up a new server from that snapshot, assign the primary IPs, and it should work immediately – no need to change any DNS settings.

Make sure to test this process a few times so you’re comfortable with it.

Conclusion and Final Thoughts

Congratulations on reaching the end!

Remember to monitor your setup, fine-tune your spam settings, and keep your SSL certificate renewed.

And just as important – test your disaster recovery plan regularly to make sure everything still works as expected. It’s better to catch issues during a test than during a real outage.

If you run into any issues or need further help, feel free to revisit this guide or reach out for assistance. Happy emailing!

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Stop Using APT! Switch to Nala

Thu, 20 Feb 2025 00:00:00 GMT

I recently started using the Nala package manager, and I was immediately impressed.

As a longtime Ubuntu fan, I was used to managing packages with APT, but Nala completely changed the game for me.

It acts as a new frontend for APT – just as APT itself is a frontend for the underlying package manager – but with a much cleaner, more readable, and user-friendly experience.

The way it simplifies package management while staying fully compatible with APT, all while adding new functionalities, makes it a must-try for any Linux user.

Why Nala

For newer Linux users, APT can often be confusing when it comes to installing or upgrading packages.

Even for me, someone who has been using Linux – specifically Ubuntu – every day for a long time, I find APT's output frustrating and unreadable.

Nala improves this experience by eliminating unnecessary messages, enhancing the formatting of package details, and using color coding to clearly indicate the actions being taken – whether it's installing, removing, or upgrading a package.

But that's not all. Nala also brings exciting features like parallel downloads , speeding up package installations, and Nala Fetch , which lets you pick the fastest mirrors.

It’s just great!

Installation

We install Nala using APT. Yeah, you heard that right – APT!

When I first installed it, I couldn’t help but wonder, Would APT be sad? It knows I'm about to replace it... but don't worry, APT’s still here to stay.

Use the following command to install Nala:

sudo apt install nala

Once Nala was installed, my next step was to explore its functionality.

The first thing I do after installing any new package (especially one I haven't used before) is check its man page or use the -h option to see what options are available and get a sense of how to use it.

So, of course, I went ahead and tried the -h option with Nala, and I was seriously impressed.

See for yourself:

Nala's help output

It is organized, put inside its own box, colored, and tells you exactly how to use it.

Nala Fetch

Before you start using Nala, I want to show you one of its best features to speed up package management: the fetch command.

Just run this:

sudo nala fetch

This basically tests all available mirrors, provides you with a list of them, tests the latency, and scores each mirror.

It then lets you pick the fastest ones to use and writes them to the nala-sources.list file inside the /etc/apt/sources.list.d/ directory.

And that's it! Now you are using the fastest mirrors.

💡

At the moment, the fetch command works only on Debian, Ubuntu, and derivatives still tied to the main repositories.

Parallel Downloads

Unlike APT, Nala supports parallel downloads, meaning it can download up to three packages at once from each mirror. This speeds up package installations significantly, especially for many small packages.

Nala also alternates between mirrors, boosting download speeds by choosing the fastest available server. If one mirror fails, Nala automatically switches to the next one, ensuring your downloads continue without interruption.

I love how Nala handles these tasks on its own, without relying on APT for downloading or verification, making it both faster and more reliable.

Running Updates

Nala provides commands to update your server, just like APT, but with an additional useful option.

Instead of running apt update followed by either apt dist-upgrade or apt full-upgrade, Nala simplifies the process with a single command:

sudo nala full-upgrade

This command updates your package lists and upgrades the server by removing, installing, and upgrading packages – all in one step.

However, if you prefer the traditional APT workflow, Nala still provides equivalent commands.

You can use these two commands to update package lists and upgrade installed packages:

sudo nala update sudo nala upgrade

These work exactly like apt update and apt upgrade, giving you the same functionality with Nala’s improved readability.

History Tracking

One of the commands Nala provides that I love is the history command!

It gives me a clear log of past package operations, like this:

ivan@vm1:~$ nala history ID Command Date and Time Altered Requested-By 1 upgrade base-files bind9-dnsutils bind9-host bind9… 2025-02-20 08:03:23 UTC 65 ivan (1000) 2 full-upgrade 2025-02-20 08:04:41 UTC 3 ivan (1000) 3 install bpytop 2025-02-20 08:50:01 UTC 2 ivan (1000)

Here’s what each column means:

ID: The operation number in chronological order.
Command: The exact package management command that was run.
Date and Time: When the command was executed.
Altered: The number of packages that were installed, upgraded, or removed.
Requested-By: The user who ran the command.

This makes it easy to track package changes on your server, ensuring you always know what was installed, upgraded, or removed.

Conclusion and Final Thoughts

With its clean and readable output, faster downloads through parallel connections, and convenient features like Nala Fetch and History Tracking , Nala offers a smoother experience compared to APT.

The rest of the commands remain the same as APT – such as install for installing packages or autoremove to remove unneeded ones – so there’s little to no learning curve for existing users.

If you found value in this reading or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Real-Time Linux Server Monitoring with dstat

Wed, 12 Feb 2025 00:00:00 GMT

Whether you're debugging an issue or optimizing server resources, real-time data on CPU usage, memory usage, disk activity, and network traffic is crucial.

This is where dstat shines – a powerful tool that provides detailed, real-time server performance insights.

In this guide, we’ll explore dstat and how you can use it to monitor and analyze your Linux server.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

What is `dstat`?

dstat is a real-time performance monitoring tool that provides a comprehensive view of server statistics, including CPU usage, memory usage, disk I/O, and network traffic.

Unlike tools like vmstat, iostat, and netstat, which focus on specific resources, dstat combines them all into a single command, making performance analysis easier.

Installation

dstat is not installed by default. To install it, use the following command:

sudo apt install dstat

Once installed, you can verify the installation by running:

dstat --version

Now that dstat is installed, let’s explore how to use it for monitoring your Linux server.

Basic Usage

Running dstat without any options defaults to the -a (all) option, which is equivalent to:

dstat -cdngy

This command displays real-time statistics for:

CPU usage (-c)
Disk activity (-d)
Network traffic (-n)
Paging activity (-g)
System statistics (-y)

To start monitoring, simply run:

dstat

This will update the server’s performance metrics every second in real time. You can stop it anytime by pressing Ctrl + C.

Here's an example of the output:

\----total-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai stl| read writ| recv send| in out | int csw 0 0 100 0 0| 0 0 |7160B 4953B| 0 0 | 167 145 0 0 100 0 0| 0 0 |1038B 812B| 0 0 | 108 102 0 0 99 0 0| 0 0 | 66B 310B| 0 0 | 33 52 ^C

The dstat output provides a quick overview of key performance metrics to help monitor server activity in real time.

CPU usage (usr sys idl wai stl) displays user and system CPU usage, idle time, I/O wait time, and stolen CPU cycles, because it is being used by another virtual machine (VM).
Disk activity (read writ) shows the total disk read and write operations.
Network traffic (recv send) indicates the amount of data received and sent over the network.
Paging (in out) represents memory pages swapped in and out.
System metrics (int csw) track system interrupts and context switches.

Additionally, you can enable time/date output with the -t option:

dstat -t

This will display the time and date alongside the performance metrics, giving you a timestamp for each data point. You can also use --time-adv for millisecond precision in the time output.

Monitoring CPU Usage

To get detailed insights into CPU performance, dstat provides options to track overall and per-core usage in real time.

To monitor CPU statistics, use the -c option:

dstat -c

This will display real-time CPU usage, including user (usr), system (sys), idle (idl), wait (wai), and stolen (stl) CPU percentages.

The usr value represents the CPU usage for running user processes (outside the kernel), while the sys value represents the CPU usage for running kernel processes. The idl value indicates the percentage of CPU that is idle. A high wai value indicates that processes are waiting on disk I/O, and stl is relevant in virtualized environments, showing CPU cycles stolen by the hypervisor.

If you want to monitor all available CPU cores dynamically, you can use:

dstat -c -C 0,1,2,3

This will display usage for all CPU cores individually, providing a detailed breakdown of CPU activity.

The -c option will ensure that only CPU performance metrics are shown, while the -C option allows you to specify individual CPU cores by their numbers. The number 0 refers to the first core, as Linux starts counting cores from zero.

You can also use the -f option for a full display of all available CPU cores, instead of specifying each core individually with the -C option.

Additionally, the --cpu-use option focuses only on CPU usage stats and provides a per-core usage breakdown:

dstat --cpu-use

This will show per-core CPU usage in a simplified format, helping you track load distribution across different cores.

To monitor the server load, use the -l option:

dstat -l

This will display a load average over different time intervals (1 min, 5 mins, and 15 mins).

You can also track process-related statistics using the -p option:

dstat -p

This displays the number of runnable , uninterruptible , and newly created processes, helping you analyze CPU load and responsiveness.

Runnable are processes ready to execute.
Uninterruptible are processes stuck waiting for I/O operations to finish. If too many processes remain in this state, it may indicate disk slowness or other bottlenecks.

For a total count of running processes, use the --proc-count option:

dstat --proc-count

This displays the number of processes, with scal showing the runnable processes and tota showing the total number of processes.

Monitoring Disk Usage

To monitor disk activity, dstat provides options that allow you to track real-time disk read and write operations. You can monitor overall disk usage or focus on specific devices and partitions to analyze disk performance more effectively.

To monitor disk activity in real time, use the -d option:

dstat -d

This will display the total read and write operations on your disk, helping you track how much data is being transferred.

If you have multiple storage devices or partitions, you can specify which one to monitor by using the -D option, followed by the device name, like this:

dstat -d -D sda

This command will display disk activity specifically for the sda device.

The -f option works here as well, providing a full display of all disks, so you don't need to specify them individually.

If you want to go deeper into I/O performance, you can use the -r option to track the number of read and write requests made to the disk. For example, if the server reads 100 files and writes 50 files in one second, this will show the number of I/O operations being initiated:

dstat -r

This will show the number of read and write requests, helping you analyze I/O activity in detail.

Additionally, the --aio option tracks asynchronous I/O operations (non-blocking operations). For example, instead of waiting for one read operation to finish before starting the next, asynchronous I/O allows multiple read operations to happen simultaneously, making the disk more efficient:

dstat --aio

These options help you monitor the frequency and efficiency of disk I/O operations in real time.

Monitoring Memory Usage

To monitor memory usage in real time, dstat provides options to track both physical memory (RAM) and swap space.

To monitor memory usage in real time, use the -m option:

dstat -m

This will display memory usage details, including used, free, buffer, and cache memory.

Buffers temporarily store data being read from or written to disk, improving disk I/O performance by reducing direct disk access. Cache holds frequently accessed files and directory data in RAM, allowing for faster retrieval without needing to read from disk repeatedly. The server dynamically manages this memory, freeing it when needed for active processes.

To monitor swap space usage, use the -s option:

dstat -s

This will display real-time swap usage, showing the amount of swap space that is free and used.

You can also use the -g option to monitor paging activity, which shows how much data is being swapped in and out of memory.

Paging refers to the process of swapping memory pages between physical memory (RAM) and disk storage (swap space) when the server is under memory pressure. This process helps manage memory when the available physical RAM is full.

Page in: When data is moved from disk (swap space) back into RAM.
Page out: When data is moved from RAM to disk (swap space) to free up memory for other processes.

This is helpful to understand memory management and identify potential performance issues.

Monitoring Network Traffic

To monitor network traffic in real time, dstat provides options to track incoming and outgoing data on your network interfaces.

You can easily track network statistics, such as the amount of data received and sent, and use various options to monitor specific interfaces or the overall network activity.

To monitor network traffic, use the -n option:

dstat -n

This will display real-time statistics for data received and sent over the network interfaces.

If you want to monitor specific network interfaces, such as eth0, or include a summary of all interfaces, you can use the -N option:

dstat -n -N eth0,total

This command will show network statistics for eth0 as well as a total summary for all interfaces combined. This helps you track both individual and overall network performance.

Additionally, to monitor the number of network packets received and transmitted instead of data volume, use the --net-packets option:

dstat --net-packets

This provides a count of packets sent and received, which is useful for analyzing network load and diagnosing packet-related issues.

Conclusion and Final Thoughts

In conclusion, dstat is a versatile tool that helps you monitor real-time server performance by tracking key metrics like CPU, disk, memory, and network activity It’s a great resource for identifying and addressing performance issues.

For more options and advanced usage, check the dstat man page:

man dstat

This will provide additional details to customize the tool for your needs.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Benchmarking Disk Performance on a Linux Server

Fri, 07 Feb 2025 00:00:00 GMT

Disk performance is a critical factor that impacts everything from boot times to database queries.

I learned this the hard way when a slow disk caused high I/O wait on my server, bringing everything to a crawl. Web pages loaded slowly, database queries took forever, and the entire server felt unresponsive.

In high-performance environments such as web servers and databases, where thousands of I/O operations occur every second, disk speed becomes a non-negotiable priority.

That's why in this guide, we'll explore how to benchmark disk performance on a Linux server using various tools and metrics, helping you assess throughput, latency, and IOPS effectively.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Author's Note

Before we jump into benchmarking, here’s an important disclaimer: Never run these tests on a production environment.

On one hand, you risk breaking things, and on the other hand, the results won’t be accurate.

For the most accurate results, follow these best practices:

Run tests on an inactive server with no active I/O operations or memory usage.
Repeat each test 2–3 times to account for variability.
Calculate the average value of your results for better accuracy.

Understanding Disk Basics

For example, let's look at hosting a WordPress website to help understand how disk performance affects website speed.

Disks are responsible for storing your website files, databases, and logs, so their speed directly influences the user experience.

Key factors that affect disk speed include the type of disk and variousdisk speed metrics.

Type of disk:

HDD (Hard Disk Drive): Slower performance due to the use of spinning mechanical disks.
SSD (Solid State Drive): Much faster than HDDs, as they have no moving parts.
NVMe SSD: The fastest type of SSD, offering superior speed compared to standard SSDs.

Disk speed metrics:

Latency: The time it takes to access data on the disk. Lower latency translates to faster data access and a quicker website.
Throughput: The amount of data that can be transferred per second, measured in MB/s. Higher throughput means faster data transfer.
IOPS (Input/Output Operations Per Second): This metric measures how many small read/write (I/O) operations can occur per second. Higher IOPS is crucial for databases, as it allows for faster processing of multiple requests.

HDDs are good at sequential reads/writes but slow at handling small random requests. SSDs and NVMe disks excel at random IOPS and low latency.

Types of Disk Operations

When benchmarking, you'll typically test two different types of read/write operations:

Sequential I/O: Large continuous files, like downloading a big backup or saving a large video.
Random I/O: Small, scattered files, like database queries or WordPress files.

For example, WordPress websites mainly depend on random read/write performance, especially for database and PHP file reads.

Basic Read/Write Tests

In the following, I’ll show you how to test your disk’s sequential read/write performance.

Sequential I/O tests measure how quickly data can be written to or read from the disk in a continuous, linear order.

These tests are commonly used for tasks like large file transfers or backups and simulate real-world scenarios where data is written continuously, without the disk needing to jump to different parts.

Sequential Read Speed Performance

When deploying a new Linux server, I always begin with a quick sequential read speed test using the hdparm command.

hdparm measures both cached and buffered read speeds, giving you an idea of how quickly your server can access data from memory and directly from the disk.

It ensures accurate measurements by flushing the buffer cache before running the test since we want to know how fast data is read from the disk itself without any buffers.

Now, to run the read speed test, use the following:

sudo hdparm -Tt /dev/sda

Replace /dev/sda with the actual device name of your target disk.

Once the test completes, hdparm will display output like this:

/dev/sda: Timing cached reads: 28238 MB in 1.99 seconds = 14180.88 MB/sec Timing buffered disk reads: 6434 MB in 3.00 seconds = 2144.09 MB/sec

Timing cached reads : Measure memory read speed, indicating how fast data is accessed from memory.
Timing buffered disk reads : Reflect the actual disk read speed, showing how fast your server can read data directly from the disk.

It’s a simple yet powerful way to get an initial sense of disk performance. I love this command!

Sequential Write Speed Performance

To measure sequential write speed, you can use the following dd command to write data to a test file:

dd if=/dev/zero of=test_file bs=64k count=16k conv=fdatasync

Here’s what each part of the command does:

if=/dev/zero : Uses a special file that generates zeroes, so no actual data needs to be read, focusing on write performance.
of=test_file : Specifies the file where data is written.
bs=64k : Sets the block size to 64KB, a common block size for disk transfers in real-world scenarios.
count=16k : Writes 1GB of data (64KB * 16,000).
conv=fdatasync : Ensures that all data is flushed to the disk immediately. Without this flag, some data might remain in memory (cached) instead of being written to the disk, which could lead to inaccurate results.

Once the test completes, dd will display output like this:

16384+0 records in 16384+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.36814 s, 785 MB/s

785 MB/s is the sequential write speed of the disk, indicating how fast the disk can write data in this test scenario.

Feel free to change the bs and count values based on your specific test scenario. For example, using bs=1M and count=1024 would write a 1GB file but with a larger block size.

Advanced Disk Performance Benchmarking

fio is one of the most powerful and flexible tools for benchmarking disk performance, particularly because it can simulate real-world workloads.

To evaluate how well your disk can handle random I/O loads, we can run a benchmark that simulates both random read and random write operations.

This is especially important for websites that serve dynamic content and rely on database access like WordPress for example.

fio is not usually installed by default. To install it, run the following command:

sudo apt install fio

Here’s how you can benchmark your disk’s random I/O performance:

fio --name=random_rw --ioengine=libaio --rw=randrw --direct=1 --bs=4k --numjobs=4 --size=1G --runtime=30 --time_based --group_reporting --unlink=1

Breakdown of the command:

--name=random_rw: The name of the test job.
--ioengine=libaio: libaio is the I/O engine that enables asynchronous operations. It's optimal for testing random I/O because it simulates a real-world server where multiple I/O operations happen in parallel.
--rw=randrw: This tells fio to perform random read/write operations.
--direct=1: This forces direct I/O, bypassing the server's cache, ensuring that the test measures actual disk performance without being influenced by cached data.
--bs=4k: Block size of 4KB is common for database workloads.
--numjobs=4: This creates 4 parallel jobs, which simulates multiple users or processes accessing the disk concurrently. More jobs will help simulate a higher load and better represent the demands on a real server.
--size=1G: Each of the 4 jobs will use 1 GB of data.
--runtime=30: This runs the benchmark for 30 seconds, allowing enough time to get a solid measurement of disk performance without excessive overhead.
--time_based: This will run the test for the duration set by --runtime, instead of running it until fio finishes reading/writing the data, even if the data isn't fully processed. This ensures that the benchmark will always run for the specified period (30 seconds in this case), regardless of whether all data has been read or written, which is particularly useful for simulating time-based loads in real-world environments.
--group_reporting: This will display the results from all jobs into a single output for easier analysis.
--unlink=1: After the benchmark, unlink deletes the test files to clean up.

Once the test completes, fio will generate a large amount of output. Pay close attention to these specific lines in the output:

read: IOPS=27.8k, BW=109MiB/s (114MB/s)(3263MiB/30001msec) lat (usec): min=55, max=5530, avg=142.94, stdev=50.93 write: IOPS=22.2k, BW=86.5MiB/s (90.8MB/s)(2597MiB/30001msec) lat (usec): min=4, max=3448, avg= 6.81, stdev= 8.16

Read performance:

IOPS : 27.8k operations per second (higher is better).
Bandwidth : 114MB/s throughput (how fast data is read).
Latency : Average time per read (142.94 µs), lower is better.

Write performance:

IOPS : 22.2k write operations per second (higher is better).
Bandwidth : 90.8MB/s throughput (how fast data is written).
Latency : Average time per write (6.81 µs), lower is better.

The disk performs well with high IOPS (27.8k for reads, 22.2k for writes), meaning it can handle many operations per second. The throughput (109 MiB/s for reads, 86.5 MiB/s for writes) indicates good data transfer speeds, which is important for handling website traffic. The latency is low, especially for writes, meaning the disk responds quickly to requests.

Conclusion and Final Thoughts

In this guide, we’ve explored the essential tools for benchmarking disk performance on a Linux server.

By measuring IOPS, throughput, and latency, you can gain a clearer understanding of how your server’s disk is performing.

Remember, disk speed plays a crucial role in server responsiveness, and investing time in disk performance analysis can lead to significant improvements in your server’s stability.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Linux Server Resource Monitoring Made Easy

Sun, 02 Feb 2025 00:00:00 GMT

Resource monitoring is a critical aspect of maintaining the health and performance of your Linux servers.

Whether you're managing a personal project or a large-scale production environment, monitoring ensures that your server runs as expected, with minimal downtime and maximum efficiency.

In this guide, we’ll explore essential commands and techniques to help you monitor your server's resource usage and fix performance bottlenecks effectively.

Core Concepts to Understand

Before diving into specific commands and techniques, it’s important to understand the key areas of Linux server resource monitoring.

Memory usage, for instance, should be closely observed to ensure there is enough available RAM. When the server runs out of RAM, it starts using swap space, which is much slower and can significantly degrade performance. While some swap usage is normal, excessive swapping indicates a need for more memory or better optimization.

Storage is another key resource to monitor. Running out of disk space can bring a server to a halt, preventing new data from being written. Common causes include growing log files or unmonitored backups, which need to be managed proactively.

CPU performance also requires attention, as an overloaded CPU can’t keep up with processing demands, leading to high load averages and overall performance issues.

Input/output (I/O) monitoring is equally vital, as excessive read/write operations or traffic flow can create bottlenecks, affecting how efficiently the server handles operations.

By monitoring these core areas, you can identify and fix bottlenecks, enabling you to focus on efficiently restoring and optimizing server performance.

`top` Command

top is one of the most widely used and long-standing commands for monitoring server resource usage in real-time. It provides a dynamic, continuously updated view of system processes and their resource usage.

Unlike static commands like ps, top is interactive, allowing users to scroll through the list of processes, filter results, and even terminate processes directly from the interface.

Simply type top in the terminal, and it will display server statistics in a structured format. You can use the arrow keys to navigate through the list. To exit, simply press q on your keyboard.

The upper half of the output presents an overview of resource usage, including server uptime, the number of active tasks, CPU usage breakdown, and memory statistics.

The lower half of the output displays a detailed list of processes in different states, sorted by default in descending order of CPU usage.

Time, Uptime, User Sessions, and Load Average

The first line in top shows the current time, followed by the server's uptime, indicating how long the server has been running since its last reboot.

top - 08:59:53 up 34 min, 2 users, load average: 0.61, 0.16, 0.05

For example, the time is 08:59:53, and the server has been up for 34 minutes. It also displays the number of active user sessions (in my case, two).

💡

If you want to get more details about the active user sessions, use the who command.

The load average at the end shows the server's workload over the last 1, 5, and 15 minutes. It's essentially a measure of how many processes are actively using the CPU or waiting for it to become available.

While a load of 1.0 would represent 100% CPU usage on single-core servers, most servers today are multi-core.

For example:

On a dual-core server, a load of 1.0 means that one core is fully utilized, leaving the other core idle, which equals around 50% CPU usage.
On a quad-core server, a load of 1.0 represents about 25% CPU usage, since only one of the four cores is in use.

It’s important to note that the load average in Linux accounts for both running and waiting processes, not just those currently executing. It represents an average value, not an instantaneous measurement.

To get a rough idea of CPU utilization, divide the load average by the number of CPU cores. While this isn’t an exact measure, it can still be quite useful.

Task Summary

The second line provides information on total tasks , as well as how many are running , sleeping , stopped , or in a zombie state.

Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie

In my case, I have 105 total processes, with 2 running and 103 sleeping.

The server tracks the state of each process, which can be one of the following:

Runnable (R) : The process is executing on the CPU or ready to run.
Interruptible sleep (S) : The process is waiting for an event to complete.
Uninterruptible sleep (D) : The process is waiting for an I/O operation to finish.
Stopped (T) : The process is stopped by a signal or is being traced.
Zombie (Z) : A terminated process whose data structures are still in memory because the parent process hasn't collected its status yet.

Processes in the R state are shown as running, those in the D and S states are shown as sleeping, and processes in the T state are shown as stopped.

The zombie value shows the number of processes in the Z state.

CPU Usage

The third line shows the breakdown of CPU usage , indicating the percentage of CPU used for different tasks.

%Cpu(s): 13.6 us, 0.3 sy, 0.0 ni, 86.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

The us value represents the CPU usage for running user processes (outside the kernel), while the sy value represents the CPU usage for running kernel processes.

Linux servers uses a nice value to set the priority of a process. A higher nice value means lower priority, and a lower nice value means higher priority. The ni value shows the CPU usage for processes with a manually set nice value.

Next is id, which represents the percentage of CPU that is idle. Then comes wa, which shows the percentage of CPU spent waiting for I/O to complete (more on this later).

hi (hardware interrupts) and si (software interrupts) refer to interrupts, which are signals that request the CPU's immediate attention.

In a virtualized environment, the st (steal time) shows the percentage of CPU that is unavailable because it is being used by another virtual machine (VM).

Memory Usage

The fourth and fifth lines display the server's memory and swap usage , showing how RAM and swap space are allocated.

MiB Mem : 7941.3 total, 7485.9 free, 383.6 used, 301.6 buff/cache MiB Swap: 512.0 total, 512.0 free, 0.0 used. 7557.8 avail Mem

The total value represents the total available memory, while free indicates the amount of unused memory. The used value shows the memory currently in use by active processes.

The buff/cache value refers to memory used by the server for buffers and caching.

The fifth line provides swap space details, showing the total, free, and used swap memory. Swap is disk space used as virtual memory when RAM is full. Ideally, swap usage should be low, as frequent swapping can slow down performance.

The avail Mem value represents the amount of memory that can be allocated to processes without causing more swapping. It’s the most important value to check when determining how much RAM is available for new processes.

Unlike the free value, which only shows completely unused memory, avail Mem also considers memory that can be quickly reclaimed from buff/cache. Since Linux dynamically manages buffers and cache, this memory is available for new processes when needed.

In short, while free memory might appear low, a large buff/cache means that memory can still be used efficiently. That’s why avail Mem is the best indicator of how much RAM is actually available.

💡

If you want to know the available RAM as a percentage, use the following command: free | grep Mem | awk '{print $7/$2 * 100 }'

Process List

Now, let's dive into the lower half of the top output, which displays a list of processes in different states, such as running or stopped, along with important details about each one.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

The PID is the process ID, a unique number that identifies each process. The USER field displays the username of the user who started the process.

The PR and NI fields represent the priority of the process. NI shows the nice value, which influences the process’s priority, while PR indicates the process’s priority from the kernel’s perspective.

VIRT represents the total virtual memory used by a process, including code, data, and swapped-out memory. RES shows the amount of physical RAM used by the process, giving a more accurate picture of actual memory consumption, excluding swapped-out memory but including shared libraries if they are loaded. SHR indicates the amount of shared memory used by the process, which can be accessed by other processes (shareable), and is a part of RES. %MEM shows the process's memory usage as a percentage of the total available RAM.

In terms of comparison:

VIRT is always greater than or equal to RES since VIRT includes potential memory usage, while RES shows actual physical usage.
RES includes SHR, so to calculate the exclusive memory usage of a process, you subtract SHR from RES.
%MEM is simply RES expressed as a percentage of total available RAM.

By understanding these metrics, you can better analyze a process’s memory usage. For example, high VIRT with low RES might suggest a process with a large address space but efficient memory use. High RES with high SHR indicates heavy reliance on shared libraries, and a high %MEM highlights processes consuming significant memory.

The S field shows the process's status, indicating whether it's running (R), sleeping (S), stopped (T), or a zombie (Z). Additionally, there is a less common state, the idle kernel thread (I), which represents a process that is not currently doing any work.

%CPU shows the percentage of CPU usage consumed by the process, while TIME+ indicates the total CPU time the process has been running since it started.

Finally, the COMMAND field displays the name or command of the process, allowing you to easily identify it.

Usage Examples

So far, I have only talked about the top output and explained what it provides. However, there's much more to do.

You can manage processes and make changes directly while top is running, or you can use it with various options to customize the output and make it more suited to your needs.

For example, while top is running, you can kill a process simply by pressing k , typing the process ID, and pressing the ENTER key. You will then be asked to specify the signal with which to kill the process. If you leave this blank, top uses the default SIGTERM signal, which allows processes to terminate gracefully. If you want to kill a process forcefully, you can type SIGKILL or use its number (9). The number for SIGTERM is 15.

💡

Keep in mind that keypresses are case sensitive, so pressing K instead of k will not work or will trigger a different action.

If you want to filter processes by user, you can use the -u option followed by a specific username, like so:

top -u

This will show you only the processes started by that user.

You can press the following keys to sort the list of processes top displays:

M to sort by memory usage.
P to sort by CPU usage.
N to sort by process ID.
T to sort by the running time.

By default, top displays all results in descending order. However, you can switch to ascending order by pressing the R key.

You can also sort the list using the -o option from the command line before running top, like this:

top -o %CPU

Change %CPU to %MEM, TIME+, or any other attribute you want to sort by.

By default, top does not show the full path to the process running or make a distinction between kernelspace processes and userspace processes. If you need this information, press the c key. Press c again to go back to the default view. Kernelspace processes are marked with square brackets around them.

You can also display threads, not just processes, by pressing the H key. Notice how the second line changes from saying Tasks to Threads, and it will now show the number of threads instead of processes.

Lastly, pressing e switches between kilobytes (default), megabytes, gigabytes, terabytes, and petabytes for process usage. After selecting a size, press W to save your preference, so top starts with your chosen size next time.

Context Switching and I/O Wait

It's important to understand how a Linux server manages processes and what I/O wait is – specifically, what the wa value in top exactly represents.

Processes on a Linux server perform either I/O-bound work or CPU-bound work (like executing arithmetic operations).

I/O-bound work refers to operations where data is read from or written to storage devices. On a server running a WordPress website, common I/O operations include:

Reading files such as PHP scripts, images, or HTML pages.
Writing logs like access and error logs.
Database operations , including reading from or writing to the site's database, especially for online stores.

When a process is performing I/O tasks, it doesn't need the CPU for that part of the task. The CPU is idle while the process waits for the I/O operation to finish.

To make efficient use of the CPU, the server switches to another process that is ready to run, allowing the CPU to be used while the first process waits. This creates the illusion of multitasking , even though the CPU is only running one process at a time. The server switches between processes so quickly that it appears as though multiple processes are running simultaneously. This process is often referred to as context switching.

I/O wait is a measure of the time the CPU spends waiting for I/O operations to complete. While the CPU is capable of processing requests quickly, it may need to retrieve data from the disk. If the disk is slow or busy, the CPU remains idle until the I/O operation finishes. This waiting time is known as I/O wait, which is shown by the wa value in top.

In a well-optimized server, I/O wait is typically low – around 0-1% – which indicates that the disk is keeping up with the CPU. However, if you see an wa value of 25% , that’s a red flag. It means a quarter of the CPU’s time is wasted waiting for I/O operations, making the disk a performance bottleneck.

For example, on a server running WordPress, high wa value can lead to slower page loads or delays in handling requests because the disk can't retrieve data fast enough. If I/O wait reaches 75% , the impact is even more severe – three-quarters of the CPU’s time is spent waiting on disk operations, which can cripple server performance.

Now, back to the fact that the server switches between processes that are ready to run while others are waiting for I/O operations to finish. Even though the server can switch to another process when one is waiting on I/O, there are scenarios where you still end up with high I/O wait. This is important to clarify, as many people mistakenly think that context switching can eliminate I/O wait time, but that's not entirely true.

If many processes are I/O-bound (such as a busy web server), there might not be enough CPU-bound processes ready to run. In this case, even though the CPU is free, all available processes are blocked, waiting for I/O to finish. The CPU ends up idle during these periods, causing the wa value to increase.

Another point is that the actual time required for an I/O operation to finish doesn’t change because of context switching. While the CPU might switch to another process, the I/O operation itself still takes a certain amount of time to complete. If this latency is high (because your disk is slow), more processes will be waiting for the I/O operation to finish, and the I/O wait will increase.

If you notice a high wa value, you should immediately investigate what's causing the issue and fix it. Identifying the bottleneck (whether it's a slow disk, high disk usage, or inefficient processes) will help restore server performance.

`iostat` Command

iostat is a powerful command that provides detailed insights into your server’s I/O performance. It offers a broader view of server-wide I/O statistics. It helps you understand the overall I/O behavior of your server.

The iostat command is part of the sysstat package, which is installed on most Linux distributions.

Running the command without any options shows also CPU information, such as the number of cores, average CPU usage, and average I/O wait. Using the -d option will display only the I/O statistics for the available disks.

I always use the command with the -d and -x options, which shows extended I/O statistics for more detailed information.

You can also specify a number after the options to set the interval in seconds at which the command will be rerun. For example, running iostat -d -x 1 will execute the command every second and show updated I/O statistics.

When investigating an I/O problem, such as high I/O wait, I always pay attention to a few key metrics in the command's output.

The first important metric I look at is %util, which shows how busy the disk is. If %util is close to 100% , it means the disk is nearly full and handling as many I/O requests as possible, which can cause performance issues. If you see %util consistently near 100%, it’s a clear sign that your disk is fully utilized, and you may need to investigate the cause of the high I/O usage or consider performance upgrades.

Next, I look at these metrics, each of which provides a different view of your disk’s performance:

r/s and w/s: These show the number of read and write requests per second. They help you understand the intensity of I/O operations (IOPS).
rkB/s and wkB/s: These measure the amount of data in kilobytes being read from or written to the disk per second. They give you an idea of throughput.
r_await and w_await: These represent the average time (usually in milliseconds) that read and write requests take to complete.

Now, let’s discuss what you can learn from these metrics:

If %util is high but the await values are low, your disk might be very efficient even under heavy load.
Conversely, if %util is moderate but r_await or w_await are high, it could be that the disk is having trouble completing each operation quickly, suggesting latency problems.
If your disk shows high rkB/s 0r wkB/s values, it means large volumes of data are being transferred, which could be normal for a heavy database server but might be a performance concern in other scenarios.

Lastly, to view the metrics in megabytes instead of kilobytes, use the -m option.

`iotop` Command

iotop is a powerful command for monitoring disk I/O usage in real-time, helping you investigate the causes of high I/O wait, such as inefficient processes that are consuming excessive I/O resources.

Similar to how top displays a list of processes along with their CPU and memory usage, iotop provides detailed insights into threads consuming the most disk operations.

Sometimes, iotop is not installed by default. If that's the case, you can install it using the following command:

sudo apt install iotop

Once installed, running iotop without any options displays a real-time list of threads along with their I/O usage.

To filter and display only active threads performing disk operations, use the -o option, like this:

ivan@vm1:~$ sudo iotop -o Total DISK READ: 1530.64 M/s | Total DISK WRITE: 11.92 K/s Current DISK READ: 1530.64 M/s | Current DISK WRITE: 154.98 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 9655 be/4 root 1530.64 M/s 0.00 B/s 0.00 % 0.05 % hdparm -Tt /dev/sda ...

Total DISK READ and Total DISK WRITE show how much data processes are asking the server to read or write. Current DISK READ and Current DISK WRITE reflect the real-time data flow to and from the disk.

These values may differ for two reasons:

The Linux kernel uses caching to speed up I/O, so data might be temporarily stored in memory instead of being immediately written to or read from the disk.
The kernel may reorder I/O operations for better efficiency.

Because of this, even if processes request a lot of I/O (shown in the Total values), the actual I/O happening at any moment (shown in the Current values) might be different.

The TID (Thread ID) is a unique identifier for the specific thread within a process that is handling I/O operations. The PRIO (priority) indicates how much priority the server assigns to that thread for execution. The USER column shows the user who owns the process.

DISK READ shows the amount of data a process is reading from the disk in real-time. Similarly, DISK WRITE displays the amount of data being written to the disk by the process at that moment. These values help you identify which processes are performing the most disk operations.

The IO> column shows the total I/O activity of a process, including both disk reads/writes and swap operations. The SWAPIN column shows the percentage of swap space the process is using, which can indicate memory pressure. Monitoring both columns helps you understand if performance issues are caused by heavy disk usage, swapping, or both.

Finally, two options I always use with the command are the -P option, which lists processes instead of threads along with their PIDs, and the -a option, which shows the accumulated I/O rate.

The accumulated value reflects the total amount of data a process has read or written since monitoring started, averaged over that entire period, instead of just showing the current moment-by-moment I/O rate.

For example, rather than displaying a spike of 10 MB/s for a few seconds followed by a drop to 2 MB/s, the accumulated value averages all the data over time, giving you a clearer, overall I/O rate.

Slow Disk Causing High I/O Wait

I want to share an experience I had with a server running a WordPress website that had a very slow disk. This was when I self-hosted WordPress on a server running the LEMP stack. The server was performing very slowly, and I needed to figure out the cause.

The first thing I did was run the top command. I noticed high load average values, even though the id value was around 80% , indicating that most of the CPU was idle. This left me confused. Why was there high load when the CPU wasn’t being heavily used?

It was a challenge to pinpoint the issue, but then I noticed that the wa (I/O wait) value was high. I can't remember the exact value, but it was around 8%. Typically, the wa value should be close to 0% , and a consistent value above 1% often indicates an I/O problem.

The server had many cores, so the top command only showed average values for the %Cpu(s) line. I pressed 1 to expand the view and see the wa value for each individual core. Some of them had I/O wait values around 30% and 50%, which clearly pointed to an I/O issue.

To investigate further, I used the iostat command to check the I/O statistics and noticed that the %util value was at 100% , meaning my disk was fully utilized.

Next, I ran the iotop command to see if any processes were causing the problem, but there was no indication of a specific process to blame. I ran iotop with the -a option to get accumulated values over time.

After a while, I observed that the Total DISK WRITE value was quite low, even though the disk was fully utilized. This told me that the disk itself was slow and needed to be replaced.

From this experience, I also learned an important lesson: never save website cache to disk. Instead, use cache on RAM, which is faster and reduces I/O operations, especially if you have enough RAM. In my case, the FastCGI cache stored on the disk was the major contributor to the issue, as my disk was too slow. This was a key factor in causing the performance problem.

`df` Command

The df command is used to display the available and used disk space on filesystems. It provides an overview of how storage is allocated across the different mounted partitions of your server.

Disk space is shown in 1K blocks by default. If you want to run the df command in its human-readable format, use the -h option. This will display the sizes in a more understandable format (KB, MB, GB), making it easier to interpret.

If you don’t include a specific mount point, the command shows information on all currently mounted filesystems, with output like this when using the -h option:

Filesystem Size Used Avail Use% Mounted on tmpfs 795M 984K 794M 1% /run /dev/sda 157G 3.0G 146G 3% / tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 795M 12K 795M 1% /run/user/0

In the output of the df -h command, each column represents important information about the server’s mounted filesystems.

The Filesystem column lists the mounted devices or partitions, like /dev/sda for physical disks and tmpfs for temporary memory-based storage.

The Size column shows the total size of each filesystem, while the Used column indicates how much space is currently used. The Avail column shows the remaining available space, and the Use% column displays the percentage of space used. The Mounted on column tells you where each filesystem is mounted, like /dev/sda at the root (/) or tmpfs at /run.

We can use the -i option to check inode (index node) usage. This will display the number of inodes used and available on each mounted filesystem. You can combine it with the -h option to make the output more human-readable, like this:

Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 993K 646 993K 1% /run /dev/sda 9.9M 133K 9.8M 2% / tmpfs 993K 1 993K 1% /dev/shm tmpfs 993K 3 993K 1% /run/lock tmpfs 199K 33 199K 1% /run/user/0

The Inodes column shows the total number of inodes available on each filesystem. IUsed indicates how many inodes are currently in use, while IFree displays the remaining available inodes. IUse% represents the percentage of inodes used.

For example, /dev/sda has 9.9 million inodes, with 133K used (2%), leaving 9.8 million free. tmpfs filesystems have significantly fewer inodes in use, as they primarily store temporary files.

`tmpfs` Filesystems

The tmpfs filesystems you see in the df -h output are created automatically by the Linux server during boot. These are virtual filesystems that reside in RAM and are used for various purposes.

/run is used for storing runtime data for processes, such as PID files and socket files. It’s similar to /tmp, but the data in /run doesn't survive a reboot.
/dev/shm is used for Inter-Process Communication (IPC) , allowing multiple processes to share data in memory. This is faster than writing to disk.
/run/lock is used for lock files, ensuring that certain resources are accessed by only one process at a time.

The size of /dev/shm is usually set to half of the server’s available RAM. You might think this is a waste, especially if it’s not used. However, the memory for /dev/shm isn’t pre-allocated, and processes can use it if needed. The size of /dev/shm represents the maximum memory it can use, not what’s reserved. The server gives priority to active processes, so if other processes use 80% of the RAM, /dev/shm will only use the remaining 20%.

Sometimes, even though the server gives priority to active processes, a poorly behaving or unexpectedly resource-consuming process might end up filling up /dev/shm, leaving other processes waiting for memory to become available. In such situations, the server will start using swap memory to compensate for the lack of available RAM.

Once the swap is fully used, the Out-of-Memory (OOM) Killer is invoked to free up memory by terminating one or more processes. It doesn’t prioritize cleaning out /dev/shm, so important processes may be killed, disrupting services.

We want to avoid triggering the OOM Killer whenever possible. It doesn't happen often, but it's good to keep it in mind.

For this reason, it's a good idea to keep an eye on how much space (RAM) /dev/shm is using.

`du` Command

The du command provides information about how much space individual files and directories consume. Unlike the df command, which summarizes entire filesystems, du is useful for drilling down to find large files or directories consuming excessive space.

Running the command alone will produce this output:

4 ./.config/procps 8 ./.config 4 ./.cache 36 .

The du command displays the size of each directory and subdirectory, with the numbers representing disk usage in kilobytes. The final line provides the total disk usage for the directory where the command was executed, which in this case is 36 KB.

However, the standard du command does not specify the unit of measurement, which can make the output more difficult to interpret. Additionally, it does not list the size of individual files inside the directory by default, and only includes their total size in the summary at the final line.

We can use the -h and -a options to make the output human-readable and list the sizes of all files, like this:

4.0K ./.config/procps 8.0K ./.config 4.0K ./.bashrc 4.0K ./.profile 4.0K ./.bash_logout 4.0K ./.bash_history 4.0K ./.lesshst 0 ./.cache/motd.legal-displayed 4.0K ./.cache 36K .

In this output, the -h option makes the sizes readable (KB, MB, GB), and the -a option lists the size of every file, not just the directories.

If you only want to see the total disk usage of a directory, use the -s option along with the -h option, like this:

36K .

A particularly useful flag that I especially love is the --time flag. It shows the time of the last modification for any file within the directory or subdirectory. I often use this flag in conjunction with the -a and -h options.

The last option I want to talk about is the -d option. This option allows you to specify the depth of directories to display, making it useful when you only want to see the sizes of top-level directories without drilling down into every subdirectory.

For example, using du -d 1 -h will show the total size of each directory at the first level, providing a clearer overview of disk usage without excessive detail.

`free` Command

The free command is one of the most widely used command to display server memory statistics. The information it presents is similar to what the top command shows about memory usage, but in a more concise format.

Running the free command with the -h option provides the output in a human-readable format, making it easier to understand by displaying memory sizes in KB, MB, or GB, like this:

total used free shared buff/cache available Mem: 7.8Gi 384Mi 7.3Gi 988Ki 301Mi 7.4Gi Swap: 511Mi 0B 511Mi

The server has a total of 7.8 GB of RAM. Out of that, 384 MB is in use, and 7.3 GB is free. It has 301 MB in cache or buffers that can be freed if needed, and 7.4 GB of memory is still available for use by new processes. The Swap section shows 511 MB of swap space, which is completely free since no swap has been used.

One important clarification: the shared value in the free command output is not the same as the SHR value in top. In free, shared refers to the total memory used by all tmpfs filesystems (such as /dev/shm). In contrast, SHR in top represents the amount of potentially shareable memory a process is using, such as shared libraries or memory-mapped files. This does not mean the memory is actively shared – only that it could be if other processes use the same resources.

Remember when I mentioned keeping an eye on/dev/shm to monitor how much space (RAM) it’s using to prevent the OOM Killer from being invoked? You can use the free command and check the shared value instead of using the df command for this.

`uptime` Command

The uptime command displays how long the server has been running since its last reboot. It also shows the current time, the number of logged-in users, and the load average over the past 1, 5, and 15 minutes.

I previously talked about load average, which indicates how much work the CPU is handling and whether the server is under heavy load.

Running the uptime command alone will produce output similar to this:

12:34:56 up 10 days, 4:22, 2 users, load average: 0.45, 0.30, 0.25

Here, 12:34:56 represents the current time, up 10 days, 4:22 means the server has been running for 10 days and 4 hours 22 minutes since its last reboot, 2 users shows the number of logged-in users, and the load average values represent the server load.

The -p option displays the uptime in a simpler, more human-readable format:

up 10 days, 4 hours, 22 minutes

The -s option shows the exact date and time when the server was last booted:

2025-01-20 08:12:34

These options provide more flexibility when checking server uptime.

Conclusion and Final Thoughts

In this guide, we've explored various commands to monitor and troubleshoot resource usage on Linux servers, with a focus on CPU, memory, storage, and I/O.

Understanding how your server manages resources and identifying potential bottlenecks is crucial for maintaining good performance.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Why Linux Servers Need Reboots and How to Avoid Downtime

Fri, 17 Jan 2025 00:00:00 GMT

Rebooting a Linux server might seem straightforward, but in a production environment, it requires careful planning to avoid downtime or disruptions.

Technologies like live patching can delay the need for a reboot and minimize downtime, but reboots are unavoidable at times.

While I can’t promise you’ll never need to reboot again after reading this, I will help you do so with confidence and minimal impact when necessary.

This guide explains why reboots are sometimes required, how to handle them correctly, and how to avoid them whenever possible.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Why Does Linux Ask for a Reboot?

While Linux is known for its stability and can run for extended periods without any issues, there are specific scenarios where a reboot becomes necessary.

Here are the most common reasons:

New Kernel
1. Running updates often involves installing a new kernel version, which requires a reboot to load. This is one of the most common reasons for needing a reboot.
Critical Updates
1. Some security updates are not kernel-related but still require a reboot to be fully applied.
2. Essential components, such as the glibc library or critical drivers, often require a reboot to take full effect.
Filesystem Changes
1. Significant changes to the filesystem, such as modifying the /etc/fstab file, sometimes necessitate a reboot to ensure the server applies the new configurations correctly.
Hardware Changes
1. Adding or replacing hardware components, such as CPUs, GPUs, or memory, usually requires a reboot to take effect.
Configuration Changes
1. Modifications to key system settings, such as bootloader configurations (GRUB) or kernel parameters, often require a reboot to apply.
Unexpected Crashes or Freezes
1. Issues like kernel panics, hardware faults, or other unexpected failures may make a reboot necessary to restore functionality.

These are some of the most common reasons, but you may face other situations as well that require a reboot while managing Linux servers.

Checking If a Reboot Is Necessary

Determining whether the server requires a reboot or not is your first step.

One simple way to check is by looking for the presence of the /var/run/reboot-required file using the following command:

sudo cat /var/run/reboot-required

This file is a signal used on Debian-based distributions, like Ubuntu, to indicate that a reboot is needed after certain package updates. If the file does not exist, it typically means no reboot is currently required.

Another method is to use the needrestart command:

ivan@vm1:~$ sudo needrestart [sudo] password for ivan: Scanning processes... Scanning linux images... Running kernel seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. ivan@vm1:~$

This command examines running services, libraries, and kernel modules to determine if any of them need to be restarted or if a reboot is required.

If your server needs a reboot, you may also see a message every time you SSH into the server, notifying you that a reboot is required.

Avoiding Downtime

In the following sections, I’ll show you some ways to avoid downtime and accomplish tasks without a reboot.

These are just examples of how I would handle these situations, but there may be cases where a reboot is unavoidable or where these methods might not work as expected.

Skipping Kernel Reboots

As I mentioned earlier, some updates involve installing a new kernel version, which often includes important fixes for vulnerabilities in the older version.

To load the new kernel and eliminate the security risks, a reboot is required. Without it, your server would continue using the older, vulnerable kernel.

However, you can use a live patching service, which allows you to patch the running kernel in real-time without needing a reboot. While the kernel image stored on the disk (the one used during boot) remains unchanged, the running kernel is patched, closing the vulnerability.

This can buy you time and help you avoid an immediate reboot, but eventually, you’ll still need to reboot to update the kernel image on the disk.

It’s also important to note that new kernel releases often come with new features and enhancements, which you won’t benefit from until a reboot is performed.

Managing Critical Updates

Some security updates are not related to the kernel, and updates to essential components, such as the glibc library, may still require a reboot to be fully applied.

In some cases, a reboot is needed because certain services must be restarted to reflect the updates. However, instead of rebooting, you can identify and restart only the affected services manually.

You can use the needrestart command, which I covered earlier, to identify which services need a restart after an update.

Once identified, you can restart these services individually, avoiding the need for a full reboot.

Filesystem Changes

If you make any filesystem changes, you could try remounting instead of rebooting.

Remounting is the process of making a filesystem or partition accessible again after it has been modified (for example, after changes to its mount options). This can be done without requiring a reboot.

Let me demonstrate that by changing a mount option for the root / partition.

First, let me check which mount options are specified for the root partition in the /etc/fstab file:

ivan@vm1:~$ sudo cat /etc/fstab [sudo] password for ivan: ... # UUID=93050c6c-d74b-9257-75fb-27312cd730f2 / ext4 errors=remount-ro 0 1 ... ivan@vm1:~$

As you can see, we have the errors=remount-ro mount option specified for the root partition.

Now, I will add noatime to it, which will prevent the access time of files from being updated on every read, like this:

UUID=93050c6c-d74b-9257-75fb-27312cd730f2 / ext4 errors=remount-ro,noatime 0 1

Before remounting, let's run the mount | grep ' / ' command to see the current mount options for the root partition:

/dev/sda on / type ext4 (rw,relatime,errors=remount-ro)

As you can see, currently, we have the relatime option enabled.

Next, let’s try remounting the root partition to apply the changes without rebooting:

ivan@vm1:~$ sudo systemctl daemon-reload ivan@vm1:~$ sudo mount -o remount /

After remounting, let's check the mount options again:

ivan@vm1:~$ sudo mount | grep ' / ' [sudo] password for ivan: /dev/sda on / type ext4 (rw,noatime,errors=remount-ro) ivan@vm1:~$

As you can see, the noatime option has now been applied, and the relatime option has been replaced.

Tweaking Kernel Parameters

Sometimes, we need to tweak kernel parameters, such as when hardening the Linux kernel.

This typically involves adding the desired parameters with their modified values to the /etc/sysctl.conf file and rebooting the server for the changes to take effect.

However, instead of rebooting, you can apply the changes in real-time by running:

sudo sysctl -p

This command reloads the parameters from the sysctl.conf file without requiring a reboot.

Alternatively, you can use the following command to modify a parameter temporarily:

sudo sysctl -w [parameter]=[value]

Keep in mind that changes made using sysctl -w will only persist for the current session and will be reset upon reboot.

That’s why it’s best to add your parameters to the sysctl.conf file, apply the changes in real-time, and ensure they remain in effect even after the server reboots.

Reboot After Kernel Panic

Kernel panics can result in your server becoming unresponsive and require a reboot to recover.

To prevent downtime due to a kernel panic, you can configure the server to automatically reboot after a panic occurs. This can be done by adjusting the kernel.panic kernel parameter.

Open the /etc/sysctl.conf file and add the following line at the end of the file:

kernel.panic = 10

This parameter ensures that the server will automatically reboot 10 seconds after a kernel panic occurs, minimizing downtime.

After making this change, you can apply it by running:

sudo sysctl -p

This way, if a kernel panic happens, the server will not stay down for too long and will automatically reboot, ensuring higher availability.

Different Ways To Reboot

In the following sections, I’ll walk you through several methods to reboot a Linux server, depending on the situation and the level of control you need.

These methods range from graceful reboots to more forceful options for when the server becomes unresponsive.

It’s important to choose the right method based on your server’s state and the tasks at hand.

The `reboot` Command

The reboot command is one of the simplest and most common ways to restart a Linux server.

Just type the command as is:

sudo reboot

It performs a graceful reboot by sending signals to terminate all processes and safely restarting the server.

Use the reboot command for routine restarts in a production environment or when you need a safe and straightforward reboot option.

Reboot Using`shutdown`

The shutdown command can also be used to restart a Linux server. It allows you to schedule a reboot and notify users about the upcoming restart.

Use this command for an immediate reboot:

sudo shutdown -r now

The -r option is used to indicate a reboot.

You can also schedule a reboot with a delay, like this:

sudo shutdown -r +5 "Server will reboot in 5 minutes"

It performs a graceful shutdown by notifying users and stopping services before restarting the server.

Use the shutdown command for scheduled reboots or when you want to give users a heads-up before the server restarts.

Reboot Using `init` or `systemctl`

You can reboot a Linux server using either init or systemctl, depending on the system's initialization method.

The init command is used to switch between runlevels, and runlevel 6 triggers a server reboot by reinitializing all processes. It’s mainly used on older systems with SysVinit, and while it can be useful for advanced users, it should be used cautiously in production environments.

The command for a reboot using init is:

sudo init 6

When you use init 6, all running processes are stopped and the server is restarted.

However, this approach might not always stop certain processes gracefully, making it a less ideal choice for production servers unless absolutely necessary.

For modern servers that use Systemd, the systemctl command provides more control over the reboot process.

Using the sudo systemctl reboot command ensures that systemd-managed services are stopped gracefully before the reboot, making it a safer and more reliable option for newer distributions.

Reboot via SSH

You can reboot a Linux server remotely using SSH without needing physical access.

This is especially useful in situations where the server becomes unresponsive, and a reboot is required to restore functionality.

To reboot a server via SSH, simply connect to the server using your SSH client and run the reboot command, like this:

ssh [user]@[server-ip] "sudo reboot"

Instead of using the reboot command, you could also run any other reboot command, such as shutdown or systemctl.

Reboot Through Server Provider

If you are managing a server from a server provider, like Hetzner, you should have an option to reboot the server from the cloud dashboard. Some providers offer two reboot options: a soft reboot and a hard reboot.

👉

New to Hetzner? Use my link to get free credits!

A soft reboot initiates a restart through the system, allowing it to safely stop services and processes, similar to running the reboot command.

In contrast, a hard reboot is essentially a forced power cycle, which can be used when the server is unresponsive and cannot be rebooted gracefully.

A hard reboot can sometimes result in data loss or file system corruption, so it should be used cautiously.

💡

Hard and soft reboots are more commonly available for dedicated physical servers. For virtual private servers (VPS), the options are typically limited to a soft reboot.

How to Reboot Safely

Rebooting a Linux server in a production environment requires careful preparation to minimize disruption.

The first and most critical step is to notify all stakeholders about the reboot. For logged-in users on the server, you can use the wall command to broadcast a system-wide message, like this:

sudo wall "Server rebooting at 4 AM."

This ensures that users are aware of the reboot and have time to save their work.

If you’re running a hosting business or managing services for clients, it’s equally important to notify customers and anyone else who could be affected by the downtime.

Next, it’s essential to verify the server’s health. Use commands like top, df -h, and free -m to check CPU, memory usage, and disk space.

Tools like htop provide a more detailed, real-time overview of the server, helping you identify any resource bottlenecks.

These checks provide an overview of the server’s performance and help identify any issues that might need attention before the reboot.

Before proceeding with the reboot, stop any critical services gracefully.

Services such as web servers (Apache or Nginx) and databases (MySQL) should be stopped manually to prevent data corruption or other issues. For example:

sudo systemctl stop nginx sudo systemctl stop mysql

Keep in mind, however, that some reboot commands we’ve covered – like reboot and shutdown – already stop services gracefully as part of the reboot process. If you’re using one of these commands, this step may not always be necessary, but it’s a good habit to verify and manually stop any especially critical services beforehand.

Step four is to ensure you have up-to-date backups available. Before rebooting, always verify that critical data and configurations are backed up in case anything goes wrong during the reboot process. This precaution will help you recover quickly if needed.

Finally, proceed with the reboot using your preferred method. Once the server restarts, verify that all services have come back online and are running as expected. Check the status of essential services such as Apache, MySQL, or Nginx, and inspect the logs for any errors that might have occurred during the reboot process.

Conclusion and Final Thoughts

Rebooting a Linux server, especially in production, is not a task to take lightly.

By understanding why reboots are necessary, planning thoroughly, and leveraging alternatives where possible, you can minimize downtime and ensure a smooth process.

👉

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Auditing Linux Servers with Auditd

Mon, 30 Dec 2024 00:00:00 GMT

Monitoring and securing a Linux server is essential for administrators who want to protect their servers from unauthorized access, suspicious activities, or unintended changes.

This is where Auditd , the Linux Audit Daemon, comes into play. It is a powerful tool that allows you to track system events, monitor file access, and keep a detailed record of what happens on your Linux server.

In this guide, you will learn what Auditd is, how it works, and how to use it effectively to audit and secure your Linux servers.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

What is Auditd?

Auditd is the user-space part of the Linux Auditing System , a built-in feature of the Linux kernel. It is a tool that helps you monitor and log events happening on your Linux server, making it easier to respond to and investigate security incidents.

Based on pre-defined rules, Auditd generates log entries to record as much information about the events that are happening on your server as possible.

Auditd can record important actions, like who accessed a file, which commands were run, or if someone made changes to critical system configuration files.

It works by listening for specific events triggered by the Linux kernel, such as:

File access (read, write, execute).
System call activities (what processes are doing).
User logins and authentication attempts.
Changes to files, directories, or system configurations.

Once these events are detected, Auditd logs them into a file located at:

/var/log/audit/audit.log

For example, if you want to know who modified the /etc/passwd file, Auditd can log the who, what, and when of that event.

In simple terms, Auditd gives you the ability to watch over your Linux server, record important activities, and keep track of changes to ensure everything is secure and under control.

Auditd Architecture

Linux distributions are divided into two key areas: kernel space , which is the core of the operating system, and user space , where user-level programs (like your web server, text editor, or any software you install) and services run.

The separation between the kernel and user-level programs helps ensure high security and stability, which is one of the main reasons Linux is so reliable.

This means that user-level programs cannot directly access system resources like hardware or kernel-controlled functions. Instead, they rely on system calls (syscalls) to ask the kernel to perform tasks for them.

Auditd operates in user space, meaning it runs outside the kernel but works closely with kernel features.

The Linux Auditing System, integrated into the Linux kernel, provides a framework for tracking and logging various activities on the server, especially security-related ones.

The kernel generates auditing data and sends it to Auditd in user space for processing.

As the user-space component of the Linux Auditing System, Auditd collects and manages this data, allowing administrators to review and analyze security-related activities happening on the server.

This seamless interaction between the kernel and Auditd ensures robust monitoring and auditing capabilities built into the Linux server.

Installing Auditd

For Debian-based distributions, like Ubuntu, you can install the latest version of Auditd along with its relevant plugins by running:

sudo apt install auditd audispd-plugins

This command ensures that both the core auditing functionality and additional plugins are installed.

After installation, the auditd service (daemon) is added. It is enabled and running by default, which you can verify using the sudo systemctl status auditd.service command.

Now, let’s check whether any Auditd rules are in effect by using the sudo auditctl -l command:

ivan@vm1:~$ sudo auditctl -l [sudo] password for ivan: No Rules ivan@vm1:~$

As you see, we haven’t created any rules yet since we just installed Auditd.

The auditctl command is what we use to manage rules, and the -l option lists the rules.

Monitoring File Changes

Let's say that I want to monitor the /etc/ssh/sshd_config file for any changes.

This file is critical because it contains the configuration settings for the SSH service, which controls remote access to our server. Unauthorized modifications to it could lead to a security breach or unauthorized access.

To monitor changes to the file, we use the auditctl command to create a rule:

sudo auditctl -w /etc/ssh/sshd_config -p rwa -k sshd_changes

Here’s what each option does:

-w : This stands for "watch", and it points to the object we want to monitor. In our case, it specifies the file to watch.
-p rwa : This indicates the object's permissions we want to monitor. In our case, these are:
- r : Read access to the file (when someone reads the contents).
- w : Write permissions (modifications to the file content).
- a : Attribute changes (metadata changes such as ownership or permissions).
-k sshd_changes : Assigns a unique key (name) to the rule for easier searching in logs.

To confirm the rule was created, we list our rules again using the sudo auditctl -l command:

ivan@vm1:~$ sudo auditctl -l [sudo] password for ivan: -w /etc/ssh/sshd_config -p rwa -k sshd_changes ivan@vm1:~$

Alright, our rule is there.

Now, let's test our rule to see how Auditd behaves when we interact with the /etc/ssh/sshd_config file.

To generate a read event, we'll use the cat command to simply view the contents of the /etc/ssh/sshd_config file:

sudo cat /etc/ssh/sshd_config

Next, let's modify the file to generate a write event. This could be something simple, like adding a comment to the file:

sudo sh -c 'echo "# Comment" >> /etc/ssh/sshd_config'

Finally, let's change the file's permissions to generate an attribute change** event:

sudo chmod 600 /etc/ssh/sshd_config

This command changes the file's permissions so that only the root user has read and write access.

Now that we've interacted with the /etc/ssh/sshd_config file, we need to check the Auditd logs to verify that the events were captured correctly.

Analyzing Rule Logs

Let's analyze the logs that were generated by Auditd after we tested our rule.

We can use the ausearch command with the -i option to interpret the logs in a human-readable format and the -k option to filter by the key we assigned like this:

sudo ausearch -i -k sshd_changes

This will display all the events related to the /etc/ssh/sshd_config file that were triggered by the rule we created.

You can also use aureport to generate a summary of the generated logs filtered by the key:

sudo aureport -i -k | grep "sshd_changes"

This will give you a summarized report of the events that were captured, making it easier to review the activity related to the file.

Let's start by using the ausearch command. The first log entry you may see is something like this:

type=PROCTITLE .... : proctitle=auditctl -w /etc/ssh/sshd_config -p rwa -k sshd_changes .... type=CONFIG_CHANGE .... : auid=ivan ses=345 subj=unconfined op=add_rule key=sshd_changes list=exit res=yes

This log entry indicates that the auditctl command itself was executed, which created the rule we applied to monitor the /etc/ssh/sshd_config file. The type=CONFIG_CHANGE line indicates that the rule was successfully added, and it also logs the rule's configuration.

Next, let's take a look at the second log entry, which corresponds to our test where we used the cat command to read the contents of the file:

type=PROCTITLE .... : proctitle=cat /etc/ssh/sshd_config type=PATH .... : item=0 name=/etc/ssh/sshd_config inode=66811 dev=08:00 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD .... : cwd=/home/ivan type=SYSCALL .... : arch=x86_64 syscall=openat success=yes exit=3 a0=AT_FDCWD a1=0x7ffeb3cb07b2 a2=O_RDONLY a3=0x0 items=1 ppid=34033 pid=34034 auid=ivan uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=345 comm=cat exe=/usr/bin/cat subj=unconfined key=sshd_changes

Let’s break it down:

type=PROCTITLE : This line shows the command that was executed.
**type=PATH**: This indicates that the /etc/ssh/sshd_config file was accessed.
**type=CWD**: This shows the current working directory when the command was run, which was /home/ivan in this case.
**type=SYSCALL**: This entry shows the system call made during the command execution. The openat syscall was used to open the /etc/ssh/sshd_config file for reading (0_RDONLY). It also contains details like the user (auid=ivan), process ID, and command that was executed (cat in this case), as well as the success status of the action (success=yes).

There should be three additional log entries corresponding to the last two tests we performed.

For the write test, when we attempted to add a comment to the /etc/ssh/sshd_config file using the sudo sh -c 'echo "# Comment" >> /etc/ssh/sshd_config' command, two log entries were created:

The first log entry reflects the initial failed attempt. This entry shows the -bash command and an openat syscall with the O_WRONLY|O_CREAT|O_APPEND flags, but the operation was unsuccessful (exit=EACCES (Permission denied)) because the user Ivan (auid=ivan uid=ivan) did not have permission to modify the file directly. This happens because the redirection (>>) is handled by the shell, and without sudo, it runs as the unprivileged user.
The second log entry corresponds to the actual execution of the command using sudo sh -c. Here, the command was executed with elevated privileges (auid=ivan uid=root), and the openat syscall succeeded (exit=3), allowing the file modification to proceed.

This dual-entry behavior is a result of how the shell and redirection operate in conjunction with privilege elevation via the sudo command.

💡

auid (Audit User ID) represents the original user ID associated with the session, while uid (User ID) represents the effective user ID of the process that generated the event.

Finally, for the attribute change test, when we modified the file's permissions using the chmod command, you should expect a log entry that includes the chmod command along with a fchmodat syscall.

👉

If you're curious about the various syscalls and their details, you can check out a comprehensive list of syscalls here.

Now, let's take a look at the logs generated by the aureport command to summarize the events we captured.

Here's a sample of the report output:

1237\. 12/20/2024 11:18:49 sshd_changes yes /usr/sbin/auditctl ivan 45451 1238\. 12/20/2024 11:25:38 sshd_changes yes /usr/bin/cat ivan 45887 1239\. 12/20/2024 11:31:43 sshd_changes no /usr/bin/bash ivan 46283 1240\. 12/20/2024 11:31:45 sshd_changes yes /usr/bin/dash ivan 46284 1241\. 12/20/2024 11:31:49 sshd_changes yes /usr/bin/chmod ivan 46299

Each of these entries shows that the relevant actions (read, write, and attribute change) were successfully logged.

Let’s pipe the output of aureport into head instead of grep so that we can view the column headers:

ivan@vm1:~$ sudo aureport -i -k | head [sudo] password for ivan: Key Report =============================================== # date time key success exe auid event =============================================== ... ivan@vm1:~$

The status in the success column will be either yes or no, depending on whether the user successfully performed an action that triggered a rule violation.

If the event wasn’t caused by a rule being triggered, the status may appear as a question mark.

You may see multiple log events when opening or editing a file with a text editor because the various syscalls made by the editor are logged individually by Auditd.

Monitoring Directory Changes

In addition to monitoring individual files, Auditd allows you to monitor entire directories for changes. This is useful for tracking activities like the creation or deletion of files, changes to file attributes, and any modifications within sensitive directories.

Imagine a team of administrators needs a secure directory called secureadmins to store critical configuration files that should only be accessible to the administrators Ivan and Elie. This directory must be highly secure to prevent unauthorized access.

The commands we are going to use ensure that only the secureadmins group can access the directory.

First, create a group for the admins:

sudo groupadd secureadmins

After that, add users Ivan and Elie to the group:

sudo usermod -a -G secureadmins ivan sudo usermod -a -G secureadmins elie

Then, create the directory:

sudo mkdir /secureadmins

Next, set the owner and group for the directory:

sudo chown nobody:secureadmins /secureadmins

This assigns the ownership of the directory to:

Owner : nobody, which prevents any specific user from having special control.
Group : secureadmins, so group members (Ivan and Elie) can access it.

Finally, set directory permissions:

sudo chmod 3770 /secureadmins

These permissions ensure the following:

The SetGID ensures that files created in the directory inherit the group ownership.
The Sticky Bit prevents users from deleting or renaming files they don’t own.
Only members of secureadmins can access, modify, or create files in the directory.
No access for anyone outside the group.

You can confirm the directory’s permissions with this command:

ls -ld /secureadmins/

The output should look like this:

drwxrws--T 2 nobody secureadmins 4096 Dec 27 10:00 /secureadmins/

Only Ivan, Elie, and processes explicitly running as part of the secureadmins group can access and modify files in the /secureadmins directory.

Now, to monitor the /secureadmins directory, run the following command:

sudo auditctl -w /secureadmins/ -k secureadmins_monitor

This time, I left out the -p option because I want track all actions (read, write, execute, and attribute changes) within the /secureadmins directory.

However, if I only want to monitor for when someone tries to cd into the directory, I would use the following command:

sudo auditctl -w /secureadmins/ -p x -k secureadmins_monitor

Now, perform actions like creating, modifying, or deleting files in /secureadmins/ as one of the group members to generate audit events. You can also change the directory's attributes, like permissions, to test attribute change logging.

Use the ausearch command to review logs for the secureadmins_monitor key:

sudo ausearch -i -k secureadmins_monitor

Or generate a summary of the logged events using the aureport command:

sudo aureport -i -k | grep "secureadmins_monitor"

If you see the root user when using theaureport command, it's because you switched to the user using the su command from root, instead of opening a new SSH session and logging in directly as that user.

Auditing Syscalls

Now, let's say I want to monitor when someone performs a certain action. To do this, I need to use syscalls.

💡

A syscall happens whenever a user issues a command that requests the Linux kernel to perform a task, as I mentioned in the Auditd Architecture section.

This isn't difficult, but the syntax is a bit trickier compared to monitoring files or directories.

With this rule, I want to monitor whenever Ivan attempts to open or create a file:

sudo auditctl -a always,exit -F arch=b64 -F auid=1001 -S openat -k monitor_ivan

Here’s what each option does:

**-a always,exit**: It tells Auditd to always log the event when the specified syscall happens, and to do so on exit, meaning when the syscall finishes.
-F : It is used to build a rule field, and we can see two rule fields in this command:
- **arch=b64**: This first rule field specifies the server's CPU architecture. b64 means that the server is running on a 64-bit x86_64 architecture. For 32-bit servers, use b32, but b64 is what you will mostly see nowadays.
- auid=1001 : This second rule field specifies the user ID number of the user that we want to monitor.
**-S openat**: This specifies the syscall to monitor. In this case, it is the openat syscall, which is used to open or create a file.

💡

You can use the id command to find the auid of a specific user.

Now, I will access the server as the user Ivan and run the cat command on the /etc/ssh/sshd_config file. Before doing so, I want to check if Auditd has already generated any logs. To do this, I use the command:

sudo ausearch -i -k monitor_ivan

To my surprise, I see numerous log entries, even though Ivan hasn’t done anything on the server yet.

In the second log entry, I see that Ivan accessed the /etc/passwd file.
In the third log entry, I see that Ivan accessed the /etc/login.defs file.
Ivan also accessed the /etc/group file.

However, Ivan didn’t choose to access these files directly. These actions were performed automatically by the server during the normal login process.

If I now use the cat command to see the content of the /etc/ssh/sshd_config file, it will generate even more log entries because running this command requires the server to load additional files in the background.

This highlights an important point: when creating Auditd rules, you need to carefully consider what you want to monitor. If your rules are too broad, you might end up with an overwhelming number of log entries, making it difficult to find the information you actually need.

To effectively monitor someone’s actions, you should create more specific rules to avoid collecting excessive, irrelevant information.

Making Rules Persistent

By default, any rule we add from the command line will only persist until we reboot the server. Once we reboot the server, these rules will be lost and the monitoring will stop.

This is because the auditctl command modifies the runtime configuration of Auditd, meaning the changes are active for the current session but do not persist after a reboot.

To ensure rules persist after a reboot, we need to save them to a rule file inside the /etc/audit/rules.d/ directory. For example, create a file called custom.rules and add our previous rule like this:

-w /secureadmins/ -k secureadmins_monitor

Next, restart the auditd service to apply the rule:

sudo systemctl restart auditd.service

Restarting the auditd service inserts all rules into the audit.rules file located in the /etc/audit/ directory. Whether the rules are in a single file or spread across multiple files in /etc/audit/rules.d/, Auditd reads them all.

If you examine the /etc/audit/audit.rules file after restarting, you'll find the rules appended at the end, like this:

## This file is automatically generated from /etc/audit/rules.d -D -b 8192 -f 1 \--backlog_wait_time 60000 -w /secureadmins/ -k secureadmins_monitor

Here’s what each part of the file means:

**-D**: Deletes all current rules to start with a clean slate. This means any rules added via the command line will be removed during the service restart (or reboot) since restarting it will read the audit.rules file again.
-b 8192 : Sets the size of the audit backlog buffer. A larger buffer prevents event loss during high log generation. If the buffers fill up, the server stops generating audit events.
--backlog_wait_time 60000 : Sets the backlog wait time to 60 seconds. This determines how long a process waits for buffer space before dropping audit events.
-f : Defines the failure mode for critical errors:
- 0 (Silent): Ignores errors and logs them silently, prioritizing uptime.
- 1 (Default): Logs errors and continues running, useful for general-purpose servers.
- 2 (Panic): Halts the system for high-security environments, ensuring no operations proceed without auditing.

The last line in the file is our rule, added from the /etc/audit/rules.d/custom.rules file.

The first four lines in audit.rules come from the default /etc/audit/rules.d/audit.rules file. To modify these settings, such as changing the failure mode to -f 2, edit the /etc/audit/rules.d/audit.rules file and restart the auditd service.

Finally, I want to introduce another way to manage rules using the augenrules command, which reads rules from the /etc/audit/rules.d/ directory.

Let’s say you’ve just installed Auditd and added your rules using rule files in the /etc/audit/rules.d/ directory. Once your rules are in place, you want to make them persistent across reboots and apply them.

Here’s how:

ivan@vm1:~$ sudo auditctl -l [sudo] password for ivan: No rules ivan@vm1:~$ sudo vim /etc/audit/rules.d/custom.rules ivan@vm1:~$ sudo augenrules --check /usr/sbin/augenrules: Rules have changed and should be updated ivan@vm1:~$ sudo augenrules --load ... ivan@vm1:~$

First, we check for any changes. If there are changes, the command will detect them. Then, we use the --load option to apply the changes. This automatically appends any newly added rules to the end of the /etc/audit/audit.rules file. If you remove a rule and rerun the commands, it will detect and apply the change as well.

The great thing about this approach is that you can check for changes and apply them without restarting the auditd service.

Using Predefined Rulesets

In the /usr/share/doc/auditd/examples/rules directory, you’ll find several predefined rulesets. To use any of these rulesets, simply copy the relevant .rules file to the /etc/audit/rules.d/ directory.

For example:

sudo cp /usr/share/doc/auditd/examples/rules/30-pci-dss-v31.rules /etc/audit/rules.d/

Once copied, restart the auditd service to apply the new rules, or use the sudo augenrules --load command.

If you find that a particular rule doesn’t work for your setup or needs to be adjusted, you can easily modify the rule file by commenting out unnecessary lines.

Deleting Rules

Now that you know how to add rules using the auditctl command or by creating custom rule files, let’s look at how to delete rules.

If you added a rule with the auditctl command (which only lasts for the current session and disappears after a reboot), you can delete it using the same command but replacing -w with -W.

For example, to add a rule:

sudo auditctl -w /secureadmins/ -k secureadmins_monitor

To delete the same rule:

sudo auditctl -W /secureadmins/ -k secureadmins_monitor

If the rule was added using a rule file in the /etc/audit/rules.d/ directory, using the -W option will only delete the rule temporarily. It will reappear after a reboot because Auditd will read all rule files again.

To delete the rule permanently, remove the corresponding rule file from /etc/audit/rules.d/ and restart the auditd service, or use the sudo augenrules --load command.

💡

You can use the sudo auditctl -D command to delete all existing rules temporarily.

If you want to delete a rule that monitors a specific syscall, use the -d option as follows:

sudo auditctl -d always,exit -F arch=b64 -F auid=1001 -S openat -k monitor_ivan

This will delete the rule we previously created to monitor whenever Ivan tries to open or create a file.

Generating Authentication Reports

By default, Auditd logs authentication attempts and user logins, both failed and successful. This is part of its core functionality to provide a security audit trail for the server.

These logs help identify potential brute-force attacks or unauthorized access attempts.

To generate an authentication report, we simply use the sudo aureport -au command. You may get a long list of authentication attempts. I will pipe the command into tail -4 to get the last 4 events:

25462\. 12/30/2024 11:39:31 git 134.209.194.117 ssh /usr/sbin/sshd no 2468 25463\. 12/30/2024 11:40:11 esroot 65.21.109.67 ssh /usr/sbin/sshd no 2474 25464\. 12/30/2024 11:40:13 gitlab 65.21.109.67 ssh /usr/sbin/sshd no 2478 25465\. 12/30/2024 11:40:35 apache 65.21.109.67 ssh /usr/sbin/sshd no 2482

As you can see, all are failed authentication attempts that Auditd logged.

If you want to get more information about a particular event, use the ausearch -a command followed by the event number, like this:

ivan@vm1:~$ sudo ausearch -a 2468 [sudo] password for ivan: \---- time->Mon Dec 30 11:39:31 2024 type=USER_AUTH msg=audit(1735558771.846:2468): pid=2637 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:authentication grantors=? acct="git" exe="/usr/sbin/sshd" hostname=134.209.194.117 addr=134.209.194.117 terminal=ssh res=failed' ivan@vm1:~$

Ubuntu, by default, also logs authentication attempts in the/var/log/auth.log file.

Tracking User Login

Besides generating authentication reports using sudo aureport -au, you can also track user login activity with the aulast and aulastlog commands.

These utilities provide detailed insights into both historical and recent user logins, allowing you to effectively monitor and analyze login patterns on your server.

The aulast command displays a history of user login and logout events by searching through the Auditd logs.

Here's an example of the output:

ivan@vm1:~$ sudo aulast [sudo] password for ivan: ivan pts/1 92.75.35.67 Sat Jan 11 09:51 - 20:33 (10:42) ivan pts/1 92.75.35.67 Sun Jan 12 08:27 - 19:47 (11:20) reboot system boot 6.8.0-51-generic Sat Jan 11 09:17 ivan pts/1 92.75.35.67 Mon Jan 13 08:56 still logged in ivan@vm1:~$

From this output:

The user ivan accessed the server multiple times from the same IP address (92.75.35.67) on different days.
The session duration is shown in parentheses (10:42 means 10 hours and 42 minutes).
Server reboots are also logged.
A user currently logged in is marked with still logged in.

The aulastlog command shows the most recent login details for all users on the server.

Here's an example of the output:

ivan@vm1:~$ sudo aulastlog [sudo] password for ivan: Username Port From Latest root /dev/pts/0 92.75.35.67 01/13/2025 09:55:57 daemon **Never logged in** ivan /dev/pts/1 92.75.35.67 01/13/2025 10:18:21 ... ivan@vm1:~$

From this output:

root last logged in on 01/13/2025 at 09:55:57 from the 92.75.35.67 IP address.
ivan last logged in shortly after on 01/13/2025 at 10:18:21 from the same IP address.
System users like daemon have never logged in, as indicated by **Never logged in**.

This utility complements aulast by focusing on the latest activity rather than providing a full login history.

Tracing a Process

Something cool I want to show you is how to trace a process! This allows you to monitor the syscalls, file accesses, and other activities performed by a running program or command.

I absolutely love this feature because it gives me the ability to dive deep into what happens behind the scenes when a specific command is executed. You can really understand what’s going on at a low level, which is great for debugging or just satisfying your curiosity!

To do this, we’ll use the autrace command, which temporarily adds Auditd rules to track the execution of a specific command.

Before we begin, make sure there are no existing Auditd rules. That’s right – you need to remove any previously created rules for autrace to work, or else you'll encounter an error.

Here’s what I do: I keep my rule files saved and use the augenrules command to load them. When I want to use the autrace command, I first delete all the current rules with:

sudo auditctl -D

Once I'm done using autrace, I simply restart the auditd service to apply my saved rules again, or use the sudo augenrules --load command.

The reason this works is that deleting rules using sudo auditctl -D only removes them temporarily, so it’s not an issue.

Now, after deleting all the rules, I want to trace the execution of the cat command against the sshd_config file. I do this by running the following command:

sudo autrace /usr/bin/cat /etc/ssh/sshd_config

This will allow autrace to track the execution of the cat command. The command will execute normally, and you will see the content of the file.

However, at the last line, you will see something like this:

Cleaning up... Trace complete. You can locate the records with 'ausearch -i -p 34358'

This indicates that the trace has finished.

You can use ausearch to review the detailed records of the traced execution, like this:

sudo ausearch -i -p 34358

Once you're finished with the trace, simply run the sudo augenrules --load command to apply your Auditd rules again.

Auditd's Configuration File

The primary configuration file for Auditd is auditd.conf, located in the /etc/audit/ directory, which controls various settings related to how the auditd service operates.

In most cases, it’s best to stick with the default settings and adjust them only when necessary.

While there are many settings available, I'll focus on a few important ones to give you a better understanding of how to configure Auditd for your needs.

The first setting you may want to change is the log_file setting, which controls the location of the Auditd log file. By default, the logs are written to /var/log/audit/audit.log. However, depending on your needs, you might want to change this path to a different location or storage device. The log file is where all the audit events are saved.

The next three settings you might want to adjust are max_log_file, max_log_file_action and num_logs.

The max_log_file setting controls the maximum size of the log file in megabytes. Once the file reaches this limit, it triggers an action defined by the max_log_file_action setting. By default, the action is set to ROTATE, which means that when the log file reaches the maximum size, Auditd will create a new log file and keep rotating older logs. The log files are numbered, with lower numbers being older. The num_logs setting controls how many rotated logs are kept before they are discarded.

By default, max_log_file is set to 8 MB, and num_logs is set to 5. For many setups, these defaults are sufficient to maintain an effective log history without consuming too much disk space. However, if your server generates a high volume of logs, you might need to adjust these values to suit your needs.

💡

For a detailed explanation of all settings, you can refer to the Linux manual page for the auditd.conf file.

To apply any changes made to the auditd.conf file, remember to restart the auditd service using the sudo systemctl restart auditd command.

More About `auditctl`

So far, we’ve used the auditctl command to create, delete, and list rules. Now, let’s explore how you can use it to control Auditd itself.

With the auditctl command, you can check the status of Auditd and change some of its configurations directly from the command line.

I want to begin with the -s option, which displays the current status of Auditd, including its configuration details:

ivan@vm1:~$ sudo auditctl -s [sudo] password for ivan: enabled 1 failure 2 pid 75821 rate_limit 0 backlog_limit 8192 lost 0 backlog 0 backlog_wait_time 60000 backlog_wait_time_actual 0 loginuid_immutable 0 unlocked ivan@vm1:~$

As you can see, Auditd is currently enabled. You can stop Auditd by running the sudo auditctl -e 0 command and re-enable it using sudo auditctl -e 1. This provides a quick way to control Auditd.

However, keep in mind that these changes are temporary. If you reboot the server, Auditd will start again. As mentioned earlier, the auditctl command modifies the runtime configuration of Auditd, so the changes are only active for the current session.

If you want to stop Auditd permanently, you can use the systemctl command to stop and disable the auditd service.

You can also use the auditctl command to change the failure mode, buffer size, and backlog wait time.

For example, you can change the failure mode with:

sudo auditctl -f 2

Change the buffer size with:

sudo auditctl -b 4096

And set the backlog wait time with:

sudo auditctl --backlog_wait_time 30000

However, for these changes to persist after a reboot, you need to modify the /etc/audit/rules.d/audit.rules file and restart the auditd service.

The -s option also provides additional information:

The pid value shows the process number of the auditd service. If the pid is 0, it indicates that the auditd service is not running.
The lost entry tells you how many events have been discarded.
The backlog field indicates how many events are currently queued, waiting for Auditd to read them.

You can use the -v option to check the version of Auditd:

ivan@vm1:~$ sudo auditctl -v [sudo] password for ivan: auditctl version 3.1.2 ivan@vm1:~$

Finally, the -h option provides help about the usage of the auditctl command. It gives you a quick summary of the available options.

Conclusion and Final Thoughts

By now, you should have a solid understanding of how to configure and manage Auditd to meet your security and auditing needs.

However, keep in mind that while Auditd is a powerful tool for alerting you about potential security breaches, it doesn't actively harden the server against them. It’s important to combine Auditd with other security measures.

👉

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Kernel Live Patching for High-Availability Linux Servers

Mon, 09 Dec 2024 00:00:00 GMT

Kernel live patching is a popular solution for Linux servers, allowing critical kernel security patches to be applied without rebooting. This is especially useful in environments where uptime is essential.

However, the concept of live patching is often oversimplified in many guides, leaving out important details.

In this guide, I’ll take a deeper look at what kernel live patching is and how to install and use a kernel live patching service.

I’ll also highlight important considerations you should keep in mind.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Author's Note

Before diving into the details, I want to share a few thoughts.

While exploring various guides online about live patching the Linux kernel, I noticed a common theme: many of them focus on promoting the concept as a way to keep Linux servers running without reboots, often emphasizing phrases like "reboot-free kernel updates" or "reboot-free security updates".

While this is partially true, it’s not the complete picture.

Installing a kernel live patching service doesn’t mean you can ignore installing new security updates or rebooting your server, nor does it replace the need for regular security updates.

It's important to understand that live patching provides only a temporary solution by patching the running kernel.

To stay fully protected, you'll still need to update and reboot your server, and I will explain why in this guide.

What is Kernel Live Patching?

Kernel live patching is a method of applying small, targeted patches directly to the running kernel's memory without replacing the kernel binary. These patches are applied in real-time, focusing on fixing specific vulnerabilities in critical areas of the kernel.

For example, if a vulnerability is discovered in the kernel version you’re currently using, a new version with the fix will be released. Normally, you’d need to update your server and then reboot to load the new kernel. However, if you delay the reboot, your server remains vulnerable, leaving it exposed until the new kernel is loaded.

Live patching addresses this issue by patching the vulnerability directly in the running kernel, avoiding the need for an immediate update and reboot.

These patches are designed to close critical security gaps, such as those leading to privilege escalation or remote code execution. However, live patching does not include non-critical fixes, new features, or driver updates.

In simple terms, live patching is like fixing a hole in a door by covering it up without replacing the door itself. This approach keeps your server secure while avoiding the downtime caused by a reboot.

Traditional Updates

Let me start by discussing traditional updates and how you would normally update your server without kernel live patching.

When deploying a new server from a cloud provider like Hetzner, the first step after accessing the server is to run updates. This is typically done by first running the sudo apt update command, followed by either the sudo apt upgrade or sudo apt dist-upgrade command.

Using apt upgrade won’t install a new kernel version. It upgrades existing packages, but doesn’t install new packages or remove unneeded ones. This means that your current kernel will receive security patches and bug fixes, but you won’t get a newer kernel version.

If a patch is released for the current version of the kernel, running apt upgrade will handle it. But if the kernel update involves installing a newer version or additional dependencies, it won't handle it.

On the other hand, using apt dist-upgrade can install a new kernel version. It intelligently handles changes in package dependencies, including installing new packages or removing existing ones if necessary.

👉

New to Hetzner? Use my link to get free credits!

Now, let me show you how to check for security-related updates. I'll use the following command:

sudo apt -s dist-upgrade | grep "^Inst" | grep -i security

This command provides a list of available security updates.

Some updates, such as kernel-related updates, require a reboot after installation. Others, which pertain to various packages, take effect immediately without needing a reboot.

For example, on a Hetzner server running Ubuntu 24.04 version, the command returns the following kernel-related security updates:

Inst linux-modules-6.8.0-49-generic (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64]) Inst linux-image-6.8.0-49-generic (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64]) Inst linux-image-virtual [6.8.0-45.45] (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64]) Inst linux-tools-common [6.8.0-45.45] (6.8.0-49.49 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [all])

Next, I'll check my current kernel version using the uname -a command:

Linux ubuntu-4gb-fsn1-2 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

This indicates I'm running kernel version 6.8.0-45 , while the updates are for 6.8.0-49 , a newer version in the same kernel series (6.8.0). It's an incremental update, not a major version change.

Because the older version (6.8.0-45) is already installed, both sudo apt upgrade and sudo apt dist-upgrade can apply the update.

The command sudo apt upgrade typically updates packages that are already installed, replacing the older version with the newer one. In this case, it will update the kernel packages to 6.8.0-49.

However, since these updates are kernel-related, a reboot is required for the changes to take effect.

For now, I will update all available packages. After the update process was finished, I saw these lines:

Pending kernel upgrade! Running kernel version: 6.8.0-45-generic Diagnostics: The currently running kernel version is not the expected kernel version 6.8.0-49-generic.

I need to reboot the server for the new kernel to take effect, but I won't do that right now, as I want to show you what a kernel live patching service can do.

Kernel Live Patching Service

I will be using KernelCare Simple Patch from TuxCare, a kernel live patching service that costs only $2.95 per month. It provides security patches for a range of popular Linux kernels, which can be applied without rebooting the server.

When a new vulnerability is found in the Linux kernel, TuxCare creates a live patch to fix it. The patch is first tested in TuxCare’s internal server farm and then gradually moved through various testing tiers to ensure it's been thoroughly tested.

Once the patch is ready, servers using KernelCare will receive and apply the patch live.

You have other options, such as the Ubuntu Livepatch Service, but I personally prefer KernelCare. It is more affordable and works reliably. While the Ubuntu Livepatch Service is available for personal use for free, it is significantly more expensive than KernelCare for commercial purposes.

Now, access your Ubuntu server and run the following command as root to install KernelCare:

curl -s -L https://kernelcare.com/installer | bash

Then, register your key (which you received by email after purchasing and can also find in your customer dashboard) using the command:

kcarectl --register [KEY]

KernelCare will automatically check for new patches every 4 hours. For now, I'll manually check and update with this command:

kcarectl --update

I received the following output:

Downloading updates Patch level 4 applied. Effective kernel version 6.8.0-47.47 Kernel is safe

As you can see, a vulnerability was fixed in kernel version 6.8.0-47.47.

Remember, I haven't rebooted yet, so my kernel version is still 6.8.0-45 if I run the uname -a command. Although I updated to 6.8.0-49 , I didn't reboot, so the underlying kernel remains the old one. This means the update hasn't fixed the vulnerability yet.

Now, let me run the kcarectl --patch-info command:

OS: ubuntu-noble kernel: kernel-6.8.0-45.45 time: 2024-10-24 07:30:13 kpatch-name: ubuntu-noble/6.8.0-47.47/CVE-2024-45016-netem-fix-return-value-if-duplicate-enqueue-fails.patch kpatch-description: netem: fix return value if duplicate enqueue fails kpatch-kernel: 6.8.0-47.47 kpatch-cve: CVE-2024-45016 kpatch-cvss: 5.5 kpatch-cve-url: https://ubuntu.com/security/CVE-2024-45016 kpatch-patch-url: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble/commit/?id=5017a6a30cd43240c688ed996b81a5daff2a7dc9 uname: 6.8.0-47.47

KernelCare patched our kernel live against CVE-2024-45016, which was fixed in kernel version 6.8.0-47.47 for Ubuntu 22.04 servers. Since I haven't rebooted, KernelCare has protected my server live without requiring a reboot. Great!

If I had rebooted after updating all packages, the new kernel (6.8.0-49) would have been loaded, and KernelCare would not have found any CVEs because the new kernel already includes the fix for the CVE.

For more information about KernelCare, check its documentation .

Considerations

It’s true that KernelCare helped fix the vulnerability live without requiring a reboot, but the underlying kernel hasn't been updated yet. Everything is virtual, so to speak.

If I run the uname -a command, I'll still see the old kernel version in the output. However, if I run the kcarectl --uname command, I'll see kernel version 6.8.0-47.47 , which has the CVE fixed. KernelCare refers to this as the "effective kernel".

After live patching, the kernel image stored on the disk (the one used during boot) and the kernel running remain unchanged, meaning we are not fully protected. And we don't benefit from any other enhancements, such as new features or non-critical fixes.

Another thing to mention is that there are some security updates that are not kernel-related but still require a reboot to be fully applied.

For these reasons, I recommend updating the server regularly to apply both non-kernel security updates and any new kernel updates. This ensures that the kernel image stored on the disk is also updated, so when you reboot, the new kernel is loaded.

I will reboot the server and check the running kernel, as well as what KernelCare has to say.

After the reboot, I ran the uname -a command:

Linux ubuntu-4gb-nbg1-6 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

As you can see, the server is now running the new kernel.

If I run the kcarectl --update command again, I get the following output:

Updates already downloaded No updates are needed for this kernel Kernel is safe

This means there is no live patching needed because we are now running the new kernel, and there are no vulnerabilities to fix.

It is wise and important to reboot from time to time.

Conclusion and Final Thoughts

The real value of live patching is that it buys you time. It allows you to decide when to reboot while still protecting your Linux server instantly and without intervention if vulnerabilities are discovered in the kernel.

It’s a powerful tool, but one that should be used as part of a broader update and maintenance strategy.

👉

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Preventing SYN Flood Attacks on Your Linux Server

Wed, 20 Nov 2024 00:00:00 GMT

SYN flood attacks are one of those frustrating things that can quietly – or not so quietly – take your server down, fast. They work by flooding your server with a ton of fake connection requests, leaving it too busy to handle the real ones from actual users.

I ran into one of these attacks a while back while hosting a small WordPress site on a VPS. One day, the site started loading really slowly, and soon after, it became completely unreachable. Even SSH was lagging. When I checked the logs, there were tons of strange connection attempts. That’s when I realized it was a SYN flood.

In this guide, I’ll walk you through the steps I now use to prevent SYN flood attacks on my own servers. We’ll cover:

Kernel tweaks to help the server handle floods more efficiently
Firewall rules to drop suspicious connections early
How to use Fail2ban to automatically block abusive IP addresses

These steps are lightweight, effective, and won’t interfere with normal traffic. By the end, you’ll have a solid setup that helps keep your server responsive and secure.

Update (October 2025)

Since writing this guide, I’ve taken these ideas much further.

I’ve built a new, modern firewall setup – a project I call UFW+ – which expands on the same SYN flood protection concepts covered here.

The new version combines UFW , Fail2ban , and nftables into a single adaptive firewall setup that can automatically detect and block floods, scans, and spoofed traffic in real time.

👉

Read the new guide : How I Built an Adaptive Firewall Setup with UFW and Fail2ban (UFW+)

Author's Note

Before we jump into the technical stuff, I wanted to share a bit of the "why" behind this guide.

I’ve always been into Linux server security, but SYN flood attacks pushed me to dig deeper. When I first started looking into ways to prevent them, I found tons of guides online – but most were either vague, overly simplified, or just plain outdated. Some tossed out a few iptables rules and called it a day. Others mentioned tools like Fail2ban but didn’t explain how to set them up properly.

That just didn’t cut it for me.

So I decided to piece things together myself – test different approaches, break things, fix them again, and eventually land on a setup that actually works.

What you’re reading now is the guide I wish I had when I started.

To really block these kinds of attacks, you need to understand the basics of how TCP works – especially the Three-Way Handshake – and how SYN floods try to break that process.

😅

Don’t worry if that sounds technical. I’ll explain it simply as we go.

Also, it’s worth noting that while DoS attacks can be handled with the right setup, large-scale DDoS attacks are tougher to stop completely. That said, even basic protections can make your server a much harder target.

Finally, even if your hosting provider (like Hetzner) has some built-in DDoS protection, it often doesn’t catch the smaller, sneakier SYN floods aimed at your VPS.

That’s why it’s on you to add extra layers of security to your servers – to make it as difficult as possible for attackers to cause trouble.

👉

By the way, if you’re new to Hetzner, you can use my link to get some free credits and try their services risk-free!

What is a SYN Flood Attack?

To understand a SYN flood attack, it helps to first know how a normal TCP connection starts. It begins with something called a Three-Way Handshake :

The client sends a SYN packet to the server, signaling a request to connect.
The server replies with a SYN-ACK packet, acknowledging the request.
The client then sends an ACK packet to complete the handshake.

Once this handshake is complete, the connection is established, and data can start flowing.

A SYN flood attack takes advantage of this handshake process by sending many SYN packets to the server – but the attacker never sends back the final ACK. The server responds with SYN-ACKs and waits, leaving these connections "half-open".

Each half-open connection uses up server resources. When too many pile up, the server gets overwhelmed and can’t handle new, legitimate connections. In extreme cases, this can cause the server to crash or become completely unresponsive.

SYN flood attacks can come from a single device (a DoS attack) or from many devices at once (a DDoS attack), often coordinated through a botnet.

Because these connections never fully open, this type of attack is sometimes called a "half-open attack".

How I Tested Everything

To experiment safely, I set up two Ubuntu 24.04 virtual machines on my MacBook – one to play the role of the "attacker". and the other as the "victim". Both were barebones servers, perfect for simulating what might happen on a real VPS.

I installed NGINX on the victim machine to keep port 80 open, then used tools like hping3 and ApacheBench to simulate traffic. While ApacheBench is typically a web server benchmarking tool, it can also flood a server with requests to mimic what happens during an attack.

The real star here was hping3. With the command:

hping3 -S -p 80 --flood victim.ip

I was able to launch a SYN flood. The --flood flag sends a constant stream of SYN packets without completing the TCP handshake – and sure enough, it didn’t take long before the victim server started lagging, then completely stopped responding. Even my SSH connection dropped.

To observe what was happening, I ran:

tcpdump -nn port 80

Watching those packets pile up in real time really showed how quickly a SYN flood can choke a server.

I also ran ApacheBench with a high number of requests to simulate excessive legitimate-looking traffic. The server's CPU and memory spiked fast. Obviously, no real user would hit the server that hard – so it was a good baseline to test defenses.

What Didn't Work

At this point, I hadn’t applied any protection – just raw, open NGINX on port 80.

I tried enabling UFW (Uncomplicated Firewall) , but it didn’t do much. Its built-in rate limiting (ufw limit) works okay for things like SSH, but it’s far too aggressive for a public web server. You don’t want to accidentally block real users just because they refreshed a few times.

I quickly realized I needed a better solution – something smarter, more flexible, and ideally automated.

Finding What Worked

After a lot of digging and testing, I built a setup that allowed a reasonable number of SYN packets per IP in a short window – and logged or dropped any excess traffic. This gave me the control I needed to separate real users from potential attackers.

Rather than just copy-pasting firewall rules from random websites, I treated them like building blocks. I tweaked, tested, broke things, fixed them, and eventually landed on a rule set that actually worked under load.

Then came Fail2ban. I configured it to monitor logs and ban IPs that triggered too many half-open connections. With a few tweaks, it worked well alongside UFW, automatically adding malicious IPs to the block list.

Once everything looked good locally, I tested the same setup on two real VPS servers (hosted on Hetzner). The difference was night and day – even under stress, the servers stayed responsive.

❗

Note: Tools like hping3 and ApacheBench should only be used for testing in safe, isolated environments. Don’t use them on servers you don’t own or control.

Kernel Settings: Tweaks That Help (a Bit)

Before jumping into firewall rules, I wanted to see if some kernel-level tuning could help – and while it doesn’t stop a SYN flood on its own, it definitely improves how the server handles things under pressure.

Enable Syncookies (If Not Already)

This one’s usually enabled by default, but it’s worth double-checking:

sudo sysctl net.ipv4.tcp_syncookies

If the result isn’t 1, just open your sysctl.conf file:

sudo vim /etc/sysctl.conf

Uncomment or add:

net.ipv4.tcp_syncookies = 1

Syncookies help the server manage half-open connections more efficiently – not bulletproof, but useful.

Tighten Reverse Path Filtering

By default, reverse path filtering is set to loose mode (2), but switching to strict mode (1) offers better protection against IP spoofing.

In /etc/sysctl.conf, find or add:

net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1

This tells your server to drop packets that don’t come from reachable IPs, which helps filter out bogus traffic.

Optimize Connection Handling

These two extra parameters can make your server more resilient under load:

net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.tcp_synack_retries = 3

tcp_max_syn_backlog – Increases the size of the queue that stores half-open connections. 4096 is a good starting point, but for high-memory servers, you can safely go higher – up to 16384 – to handle heavier traffic.
tcp_synack_retries – Reduces how long the server keeps waiting for an ACK response. Lowering this to 3 (from the default 5) helps free up resources faster when connections don’t complete.

These tweaks help the kernel respond faster and stay available when the connection queue starts filling up.

Apply the Changes

Once you're done editing /etc/sysctl.conf, save the file and reload the settings:

sudo sysctl -p

A quick reboot after that ensures everything takes effect properly.

Understanding UFW Rule Files

Since we’ll be using UFW (Uncomplicated Firewall) to help protect the server from SYN flood attacks, it’s important to understand how UFW handles rules under the hood.

Inside the /etc/ufw/ directory, you’ll find three key files:

before.rules – for IPv4 traffic
before6.rules – for IPv6 traffic
**user.rules** – for rules you add with UFW commands like ufw allow

Here’s what matters:

The before.rules and before6.rules files are processed first , before anything you manually allow via command line. That means rules here take priority , which makes them the best place to add protections like SYN flood filtering.
The user.rules file stores your typical allow/deny commands but should not be edited directly – UFW manages this file automatically.

By understanding the order UFW uses to load rules, you’ll be better equipped to insert effective protections exactly where they’ll have the most impact.

Blocking Invalid Packets (Prepping for SYN Flood Protection)

Let’s lay the groundwork for defending against SYN flood attacks by blocking invalid TCP packets.

First, make sure UFW is enabled and that you won’t lock yourself out:

sudo ufw limit 22/tcp sudo ufw enable

💡

Heads up: The ufw limit rule helps protect SSH by rate-limiting connections – it blocks IPs that try to connect too many times in a short period, which is useful against brute-force and flood attempts.

Next, we’ll filter out shady packets – like those that don’t follow the standard TCP handshake (e.g. from Nmap or other scanners). Every legit TCP connection should start with just the SYN flag.

Here’s how to block invalid stuff early:

Open both before.rules and before6.rules
Add this section after the last COMMIT line:

*mangle :PREROUTING ACCEPT [0:0] -A PREROUTING -m conntrack --ctstate INVALID -j DROP -A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP COMMIT

Then reload UFW:

sudo ufw reload

What this does:

The first rule drops anything marked as INVALID by the kernel’s connection tracker.
The second rule drops TCP packets pretending to start a connection but missing the proper SYN flag.
We add this to the mangle table and the PREROUTING chain , so these packets are filtered as soon as they arrive – before they waste any more server resources.

This is a lightweight but powerful way to cut down on noise and bad traffic before it becomes a problem.

Exploring SYN Flood Mitigation Approaches with UFW

Now that we’ve covered the basics – including blocking obviously bad traffic – it’s time to put up a solid line of defense against SYN flood attacks.

We’ll use UFW , Ubuntu’s built-in firewall, which you’ve already enabled and configured to allow SSH traffic. Plus, dropping invalid packets in the previous step gave us a strong foundation to build on.

Since we’re using ufw limit 22/tcp for SSH, it’s already got some protection built in. That said, if you want a more custom approach, the rules we’re about to go over can apply to any TCP service , including SSH.

Before settling on a robust, all-in-one solution, I experimented with a few different ways to protect my server from SYN flood attacks using UFW.

Each approach had its strengths but also some limitations – whether it was overly broad limits, excessive logging, or lack of persistence. These trials helped me understand what worked, what didn’t, and why a more refined solution was necessary.

Below, I’ll walk you through the three strategies I tested, starting with simple rate limiting and moving toward a dynamic blacklist configuration.

After that, I’ll share the final solution that combines the best of all three.

Approach 1: Basic SYN Flood Mitigation Using `limit`

In this first attempt, I used the limit module to rate-limit incoming TCP SYN packets to port 80, protecting against SYN flood attacks.

The rules were added to the end of the /etc/ufw/before.rules file, just before the final COMMIT line:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m limit --limit 10/second --limit-burst 20 -j ACCEPT -A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m limit --limit 10/second --limit-burst 20 -j LOG --log-prefix "[UFW SYN Flood Detected] " -A ufw-before-input -p tcp --syn --dport 80 -j DROP

What these rules do:

Accept Rule: Allows up to 10 new SYN packets per second (with a burst of 20) to port 80.
Log Rule: Logs any excess packets beyond this rate for monitoring.
Drop Rule: Drops any packets that exceed the rate limit to prevent abuse.

These rules were added directly to the ufw-before-input chain, which is processed before any user-defined rules.

This chain is part of the early stages of UFW’s firewall processing, meaning these rules are applied before incoming traffic reaches any services or applications running on the server. This ensures potentially harmful connections are filtered out as early as possible.

💡

Heads up: Since these rules handle port 80 traffic, there's no need to allow HTTP explicitly via sudo ufw allow 80/tcp. The before.rules rules take precedence.

For IPv6 support, add the same rules to /etc/ufw/before6.rules, replacing ufw-before-input with the ufw6-before-input chain.

✅Result:

After saving the changes, I reloaded UFW using sudo ufw reload. I then simulated a SYN flood using hping3, and the rules worked as expected.

Excess packets were dropped, and logs filled with the defined prefix showed clear evidence of blocked attack attempts. Despite the test flood, CPU usage remained stable, with only a slight increase.

⚠️Limitation:

This approach was simple and effective, but it came with a major limitation. The limit module enforces a global rate limit, meaning if one client exceeds the threshold, all others are affected.

This isn’t ideal for production environments with multiple users, so I moved on to a more flexible solution.

Approach 2: Per-IP Rate Limiting with `hashlimit`

While the first method successfully blocked SYN flood attacks, it introduced a serious limitation: the limit module applies a global rate limit.

That means if a single IP address triggers the limit, all clients are affected – even legitimate ones. This clearly isn’t suitable for a production environment with many concurrent users.

To solve this, I replaced the limit module with the more flexible hashlimit module, which lets us apply rate limits on a per-IP basis. That way, only abusive IPs are throttled, while others continue to access the service normally.

Here are the updated rules added to the end of the before.rules file, just before the final COMMIT line:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP -A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] " -A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT

What these rules do:

Drop Rule : Drops SYN packets from a single IP if they exceed 10 per second (with a burst of 20).
Log Rule : Logs these excess packets using the prefix [UFW SYN Flood Detected] for monitoring.
Accept Rule : Allows new SYN packets that haven’t exceeded the limit to reach the server.

Key Parameters Explained:

--hashlimit-mode srcip: Applies limits individually for each source IP address.
--hashlimit-srcmask 32: Ensures the limit applies to each unique IP. If needed, this can be changed (like /24) to group IPs by subnet.

✅Result:

The server successfully blocked SYN flood attempts while allowing normal traffic to pass through. During testing with hping3, I could still access the NGINX default page via curl, confirming the server was not overwhelmed and legitimate traffic was unaffected.

⚠️Limitation:

This approach does not track or remember attackers. If an IP backs off for a moment and resumes the attack, it will be treated as a fresh connection attempt.

There's no persistence or blacklist – just real-time rate limiting.

Approach 3: IP Blacklisting with `recent` + `hashlimit`

After implementing per-IP rate limiting, I wanted to go one step further – by blacklisting IPs that continuously misbehaved.

To do this, I combined the hashlimit module with the recent module. This setup lets the firewall track and remember IPs that exceed allowed limits and temporarily block them using a dynamic blacklist.

Here are the rules I added to the end of the before.rules file, just before the final COMMIT line:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -m recent --name blacklist --set --rsource -j DROP -A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] " -A ufw-before-input -m recent --name blacklist --rcheck --seconds 300 --hitcount 1 --rsource -j DROP -A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT

What these rules do:

Blacklist & Drop Rule: Drops packets from any IP that exceeds the rate limit and adds that IP to a temporary blacklist.
Log Rule : Logs excess SYN packets for monitoring.
Blacklist Check Rule : Drops packets from IPs already on the blacklist for 5 minutes.
Accept Rule : Allows new, valid connections to port 80 that haven’t exceeded the threshold.

✅Result:

During testing, my IP was automatically blacklisted after simulating a SYN flood using hping3. While blacklisted, all packets from my IP were dropped, and I couldn't access the web server.

After 5 minutes of inactivity, access was restored – confirming that blacklist enforcement and timeout both worked correctly.

You can check which IPs are currently blacklisted using:

sudo cat /proc/net/xt_recent/blacklist

⚠️Limitation:

While this setup worked better than global limits, I quickly ran into a few issues.

Since the blacklist created by the recent module lives only in memory, all entries are wiped after a reboot. That means persistent attackers could just wait it out or come back with new IPs.

Also, because IPs are blacklisted immediately after crossing the threshold, there’s a chance of blocking legitimate users who briefly spike above the limit.

This approach definitely offered more control than before, but it didn’t scale well for handling more distributed or high-volume SYN flood attacks. It felt like a solid step forward, but not something I’d rely on as a long-term or standalone solution.

Final Solution: Combining Rate Limiting and Fail2ban

In this final solution, we combine per-IP SYN flood protection with persistent blacklisting and broader access control using Fail2ban.

The recent module we used earlier only stores blacklisted IPs in memory, so all entries are lost upon reboot. While this isn't a major issue for many setups, Fail2ban offers a cleaner and more powerful alternative by:

Persistently block IPs across reboots.
Ban IPs only after repeated violations (e.g., 5 times), rather than on the first offense.
Unban IPs manually or automatically after a set duration.

This hybrid setup gives us the best of both worlds: real-time traffic control through UFW, and intelligent, manageable banning via Fail2ban.

By configuring Fail2ban to watch /var/log/syslog for entries matching our custom log prefix, we can automatically block any IP that triggers our rules repeatedly.

This approach avoids prematurely banning legitimate users who may accidentally trigger rate limits, while still stopping persistent attackers effectively.

Install and Configure Fail2ban

Install Fail2ban with:

sudo apt install fail2ban

Now create a custom filter so that Fail2ban knows what to look for in your logs. The log prefix we used in our UFW rules earlier was:

[UFW SYN Flood Detected]

So we’ll create a filter to catch that. Go to /etc/fail2ban/filter.d/ and create a file called synflood.conf:

[Definition] failregex = .*UFW SYN Flood Detected.*SRC=.*DPT=\d+.* ignoreregex =

This regex will match log entries that look like:

[UFW SYN Flood Detected] IN=... SRC=192.168.0.1 DPT=80 ...

This works for both port 80 and 443, so you don’t need to create separate filters.

Fail2ban uses "jails" to define what to monitor, how many times to allow it, and what to do when the threshold is exceeded.

Create a local copy of the jail configuration:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

This is important because you should not modify the jail.conf file directly, as it may be overwritten during an update.

Edit jail.local, and under the # JAILS section, add:

[synflood] enabled = true filter = synflood action = iptables[type=allports, name=synflood, chain=fail2ban, protocol=tcp] logpath = /var/log/syslog maxretry = 5 findtime = 600 bantime = 86400

We are creating a new jail called synflood that will block IPs from accessing all ports on the server if they appear five times in the /var/log/syslog file within ten minutes, for a duration of one day.

As you may have noticed, I’m using iptables to block IPs instead of UFW, by adding them to a new chain called fail2ban. The reason for this is that if we use UFW, it will block IPs at the user level by adding them to the user.rules file. However, this won’t take effect because the before.rules file is processed first.

You can also block traffic only on ports 80 and 443 by using the multiport option, like this:

action = iptables[type=multiport, name=synflood, chain=fail2ban, port="http,https", protocol=tcp]

This is useful if you only want to restrict web traffic while leaving other services like SSH or FTP unaffected by the ban.

Update Firewall Rules

Next, we modify UFW’s before.rules file so that:

Fail2ban can inject blocking rules early in the firewall chain.
We can isolate SYN flood protection into its own chain.

Edit /etc/ufw/before.rules and create two new chains placing them under the # End required lines line:

:SYN_FLOOD_PROTECTION - [0:0] :fail2ban - [0:0]

Then replace/add these rules:

# Direct traffic to fail2ban first -A ufw-before-input -j fail2ban # fail2ban will insert IP block rules above this line -A fail2ban -j RETURN # Now send relevant traffic to our SYN_FLOOD_PROTECTION chain -A ufw-before-input -p tcp --syn --dport 80 -j SYN_FLOOD_PROTECTION -A ufw-before-input -p tcp --syn --dport 443 -j SYN_FLOOD_PROTECTION # SYN rate-limiting and logging (HTTP) -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] " # SYN rate-limiting and logging (HTTPS) -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] " # Allow remaining SYN packets through -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT -A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -j ACCEPT

🔍 What’s Happening Here?

First , we redirect traffic from the ufw-before-input chain to the fail2ban chain.
The rule -A fail2ban -j RETURN exits the fail2ban chain and moves traffic to the next rule in ufw-before-input, which sends it to our SYN_FLOOD_PROTECTION chain.
In the fail2ban chain, Fail2ban dynamically inserts IP block rules when it detects malicious activity.
So, UFW checks the fail2ban chain first, and if the source IP is blacklisted, it’s immediately denied access.
Once it reaches the RETURN rule, traffic is passed along to the SYN_FLOOD_PROTECTION chain, which:
- drops traffic that exceeds our rate limits,
- logs it for Fail2ban to analyze,
- and allows traffic from IPs that are behaving normally.

💡

Don’t forget to update before6.rules the same way for IPv6 support – replacing ufw-before-input with the ufw6-before-input chain.

This layered approach ensures blocked IPs are filtered before they hit your rate limit logic, making your firewall more efficient and more secure.

Reload UFW and Restart Fail2ban

After replacing/adding your firewall rules and setting up Fail2ban, apply the changes by reloading UFW and restarting Fail2ban:

sudo ufw reload sudo systemctl restart fail2ban

This ensures both your new firewall chains and Fail2ban configurations are actively enforced.

Monitor and Test Fail2ban Protection

Check the status of your synflood jail:

sudo fail2ban-client status synflood

This shows how many IPs are currently banned, total bans, and the list of blocked IPs.

Let's inspect the content of the custom chain we created:

sudo iptables -L fail2ban -v -n

You will notice two rules inside this chain:

The first redirects traffic to a new chain called f2b-synflood, which Fail2ban automatically creates to manage and organize firewall rules. This ensures that if the same chain is used for another jail, the rules remain separate.
The second rule allows any traffic not handled by the f2b-synflood chain to proceed.

❗

Note: You may not see the first rule if no IPs have been blocked yet.

To view the block rules that Fail2ban has added, examine the contents of the f2b-synflood chain using the command:

sudo iptables -L f2b-synflood -v -n

Now, if I initiate an attack using hping3, Fail2ban will detect the attack and add a rule to block my IP from accessing all ports:

Chain f2b-synflood (1 references) pkts bytes target prot opt in out source destination 1657K 66M REJECT 0 -- * * 192.168.64.7 0.0.0.0/0 reject-with icmp-port-unreachable 445K 18M RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0

And indeed, Fail2ban successfully caught me.

If I want to unblock my IP, I can simply use the command:

sudo fail2ban-client set synflood unbanip 192.168.64.7

It's simpler compared to using the recent module.

You can also check the IPs that Fail2ban has identified in the log file but has not yet blocked (because they are still under the limit) by using the sudo grep synflood /var/log/fail2ban.log command.

Key Consideration & Recommended Safeguards

Before wrapping up, I'd like to highlight an important point I mentioned earlier in this guide.

Our current setup – which limits each IP to 10 SYN packets per second with an initial burst of 20 – works well for blocking small to medium-sized SYN flood attacks coming from a few sources.

🧠

However: What happens if an attacker launches a DDoS attack using thousands of IPs, each sending packets below our limit?

Since our rate-limiting rules apply per IP address , such an attack might bypass the firewall entirely – because each IP appears "well-behaved" individually. The total volume , however, could still overwhelm your server.

I haven’t tested this large-scale scenario myself, so I can't say exactly how it would behave – but it's definitely something to consider in your threat model.

What You Can Do:

Use a server provider with built-in DDoS protection: Providers like Hetzner block malicious traffic at the network level – before it hits your server.
Enable rate limiting in NGINX: Use modules like limit_req and limit_conn to throttle request rates and concurrent connections per IP.
Use a CDN like Cloudflare: Cloudflare offers DDoS protection , WAF , and edge caching , filtering harmful traffic before it reaches your origin server.**

👉

New to Hetzner? Use my link to get free credits!

Combining these external protections with your current UFW and Fail2ban setup creates a multi-layered defense strategy that can handle both targeted and large-scale attacks effectively.

Conclusion and Final Thoughts

In this guide, we walked through multiple approaches to protect your server against SYN flood attacks – from simple firewall rules using limit, to more advanced setups using hashlimit, recent, and finally integrating Fail2ban for persistent, dynamic IP blocking.

While the process involved several steps and configurations, the end result is a much more secure server, capable of mitigating SYN flood attacks while still allowing legitimate traffic through.

👉

Looking for more? Check out my full collection of in-depth Linux server security guides.

💬 Found this guide helpful?

I'd love to hear about your experiences, questions, or ideas in the discussion section below. Your feedback not only helps improve future guides – it helps fellow admins on their own journey.

Prefer a more direct conversation? Feel free to contact me anytime.

Understanding TCP and the Three-Way Handshake

Tue, 05 Nov 2024 00:00:00 GMT

The Transmission Control Protocol (TCP) is a set of rules that helps devices talk to each other over the internet.

It’s designed to reliably send data across networks, ensuring they reach their destination successfully.

A key part of this reliability is the TCP Three-Way Handshake, a process that establishes a secure connection between two devices before data exchange begins.

In this reading, we will explore the role of TCP and the Three-Way Handshake in ensuring accurate and secure data transmission.

Additionally, we will analyze an example of HTTP traffic to illustrate these concepts in action.

The TCP/IP Model

Let’s start with the TCP/IP model, as it’s essential for understanding how TCP operates.

TCP/IP stands for Transmission Control Protocol and Internet Protocol, and it lays the foundational for network communication by explaining how data moves across networks.

The TCP/IP model is made up of four layers, each with its own role in the data transmission process: the application layer, transport layer, internet layer, and network access layer.

Source: Google's Cybersecurity Course on Coursera

TCP is responsible for establishing a connection between two devices and ensuring the reliable transmission of data, while IP manages the routing and addressing of packets.

Where TCP Fits In

In the TCP/IP model, TCP works at the Transport Layer.

Here’s how the process works from start to finish:

The Application Layer passes data, such as web requests, emails, or file transfers, to TCP, which divides it into smaller units called segments.
TCP prepares these segments for reliable delivery by adding headers with essential information, like sequence numbers, to ensure they can be reassembled in the correct order on the receiving end.

💡

A segment is the basic unit of data transmission in TCP, consisting of a header and a payload (the actual data being sent).

Once the data reaches the Internet Layer, IP takes over, adding its own headers for addressing and routing, turning these segments into packets that can be transmitted across networks.
Finally, packets reach the Network Access Layer. Here, they are wrapped with additional headers and trailers, transforming them into frames. These frames are the actual units sent over the network, whether by wired, wireless, or other means.

This layered process, from segments to packets to frames, ensures that data moves smoothly and accurately across networks, with each layer handling a specific part of the journey.

Example: Visiting a Website

Let’s say you want to visit a website from your computer, which is connected to a network set up like this: your computer connects to a switch, then to a router, followed by another switch, and finally to a server that hosts the website.

You type the website's URL into your browser and click Enter. Your computer first uses the DNS server within your network to determine the IP address associated with this URL, which is typically the server's IP address where the website is hosted. After that, your computer sends a GET request to retrieve the content of the website.

Now, a process called encapsulation occurs. The application protocol we are using is HTTPS. Within the Application Layer, the GET request is encapsulated inside an HTTPS header that informs the destination server about the type of request being made.

Let’s imagine the data (the GET request) as a piece of letter that is encapsulated inside an envelope. Each time we move down the TCP/IP model, the envelope is wrapped in another envelope with additional data.

The request moves down to the Transport Layer. Here, it is wrapped in a TCP header, which includes important information like source and destination port numbers, sequence numbers, and other control data. At this stage, the data is divided into segments.

Next, we move to the Internet Layer, where a new header is added with the source and destination IP addresses, transforming the segments into packets.

Finally, the packets arrive at the Network Access Layer, where they are further encapsulated with additional headers and trailers, turning them into frames. These frames are then sent over Ethernet cables to the switch.

The switch opens the first envelope and sees that the data should be sent to a specific MAC address. It consults its MAC address table to determine which router it should send the frames to.

Now, the process of opening envelopes is called decapsulation – it’s like opening the envelopes to check their contents.

The router receives the frames and opens the second envelope, which contains the destination IP. It examines its routing tables to determine where the data should go.

Since the switch can only see MAC addresses, the router must inform the second switch of the MAC address corresponding to the destination IP – the server hosting the website.

To find out this MAC address, the router sends an ARP request. Once it obtains the MAC address, the router re-encapsulates the packets with the new MAC addresses into frames and sends them to the switch.

The data is then transmitted through the second switch to the server. The server decapsulates the frames, removing each layer until it reveals the HTTPS message. It recognizes your GET request and prepares to send back the contents of the website.

The server then repeats the same process to send the webpage data back to your computer, encapsulating it in layers as it goes.

Why TCP is Reliable

TCP begins with a Three-Way Handshake to establish a connection between two devices:

The client sends a SYN (synchronize) packet to the server.
The server responds with a SYN-ACK (synchronize-acknowledge) packet.
The client sends an ACK (acknowledge) packet back to the server.

This process ensures both devices are ready to communicate.

💡

SYN and ACK are called flags, which are used to signify the state of a connection.

Once the connection is established, TCP begins breaking down the data into smaller segments.

Segmenting the data enables smoother transmission and prevents large chunks from overwhelming the network.

Each segment is wrapped in a TCP header that holds essential details. This header includes information such as source and destination port numbers, sequence numbers, and flags.

TCP assigns each segment a sequence number. This allows the receiver to put all the data segments back together in the right order, even if they arrive out of sequence.

As the receiver gets each segment, it sends back an acknowledgment (ACK) to the sender. This confirms that each segment has been received successfully.

If an acknowledgment is missing, the sender knows that segment may have been lost and will retransmit it.

To further ensure data integrity, TCP includes a checksum in each segment’s header to verify that data wasn’t corrupted during transmission.

If any segment is corrupted, TCP will retransmit it until it is received correctly.

These features ensure that all data reaches its destination completely and accurately.

Because of its reliability, TCP is ideal for applications where accurate data delivery is essential, such as web browsing (HTTP & HTTPS), file sharing (FTP), and email (IMAP, POP, SMTP).

The Three-Way Handshake

TCP uses this handshake process to establish a reliable connection between a client (such as your browser) and a server (like a website).

Think of it as a way for both parties to agree on how they want to communicate before the actual data transfer begins.

◆ The TCP three-way handshake: SYN → SYN-ACK → ACK — view on the site

Step-by-step breakdown when visiting a website:

Initiation (SYN) :** The browser, acting as the client, initiates the connection by sending a SYN (synchronize) packet to the server hosting the website. This packet includes a starting sequence number (e.g., X), essentially signaling, “Hey, I want to start a connection, and here’s my starting sequence number: X.”
Response (SYN-ACK) : Upon receiving the SYN packet, the server responds with a SYN-ACK (synchronize-acknowledge) packet. This response includes its own starting sequence number (e.g., Y), along with an acknowledgment of the client’s sequence number incremented by 1 (X + 1). It’s like the server replying, “I’ve received your request and here’s my starting sequence number: Y. I acknowledge your number X + 1.”
Acknowledgment (ACK) : Finally, the client replies with an ACK (acknowledge) packet to confirm receipt of the server’s SYN-ACK packet. This packet contains the server’s sequence number incremented by 1 (Y + 1), signaling, “I got your response, and now we’re synchronized and ready to exchange data.”

The packet is called a SYN packet because, during the handshake, it is necessary to synchronize sequence numbers between the client and server. The starting sequence number for both sides is known as the Initial Sequence Number (ISN) , a vital component in TCP communication.

When the client sends a SYN packet to the server, it’s essentially saying, “Hey, I’m going to start counting the data I send from this number.”

In response, the server needs to take an important action: it acknowledges the client’s sequence number by incrementing it by one in its reply. This lets the client know that the server has received its message and is ready to proceed with the connection.

Sequence numbers are chosen randomly instead of starting at zero. The client and server each select random numbers (X and Y) for the following reasons:

Preventing Confusion : If the client and server have communicated previously and start a new session, using the same sequence numbers could lead to old messages being mistaken for new ones. Random starting points help prevent this overlap and confusion.
Security : Randomizing helps protect against certain types of attacks, such as TCP spoofing, ensuring that only legitimate connections are recognized.

This synchronization is crucial because TCP ensures reliable data transfer by tracking and managing the order of packets.

By knowing the sequence numbers, both the client and server can reassemble data in the correct order and detect any missing or out-of-sequence segments.

The Handshake in Action

To illustrate the TCP Three-Way Handshake in action, let’s analyze the following tcpdump output generated from a connection to a web server on port 80 (HTTP).

I used the tcpdump -nn -S port 80 command to monitor traffic on port 80, and then executed the curl http://167.235.253.88 command to initiate a request to the web server.

This output captures the sequence of packets exchanged during the handshake and the subsequent data transfer:

13:34:12.344584 IP 92.72.34.89.26232 > 167.235.253.88.80: Flags [S], seq 3531231207, win 65535, options [mss 1412,nop,wscale 6,nop,nop,TS val 3019538245 ecr 0,sackOK,eol], length 0 13:34:12.344657 IP 167.235.253.88.80 > 92.72.34.89.26232: Flags [S.], seq 2227943007, ack 3531231208, win 65160, options [mss 1460,sackOK,TS val 1425199703 ecr 3019538245,nop,wscale 7], length 0 13:34:12.372629 IP 92.72.34.89.26232 > 167.235.253.88.80: Flags [.], ack 2227943008, win 2056, options [nop,nop,TS val 3019538274 ecr 1425199703], length 0 13:34:12.372629 IP 92.72.34.89.26232 > 167.235.253.88.80: Flags [P.], seq 3531231208:3531231285, ack 2227943008, win 2056, options [nop,nop,TS val 3019538274 ecr 1425199703], length 77: HTTP: GET / HTTP/1.1 13:34:12.372862 IP 167.235.253.88.80 > 92.72.34.89.26232: Flags [.], ack 3531231285, win 509, options [nop,nop,TS val 1425199732 ecr 3019538274], length 0 13:34:12.373363 IP 167.235.253.88.80 > 92.72.34.89.26232: Flags [P.], seq 2227943008:2227943870, ack 3531231285, win 509, options [nop,nop,TS val 1425199732 ecr 3019538274], length 862: HTTP: HTTP/1.1 200 OK 13:34:12.400041 IP 92.72.34.89.26232 > 167.235.253.88.80: Flags [.], ack 2227943870, win 2042, options [nop,nop,TS val 3019538302 ecr 1425199732], length 0

The flags are shown inside the square brackets, such as [S] for SYN, [S.] for SYN-ACK, and [.] for ACK.

Let’s first analyze the first three packets, which represent the handshake:

SYN Packet (Initiation) : At 13:34:12.344584 , the client (IP 92.72.34.89) initiates a connection to the server (IP 167.235.253.88) by sending a SYN packet from its ephemeral port 26232 to the server's port 80. This packet indicates that the client is ready to establish a connection and includes its initial sequence number (3531231207).
SYN-ACK Packet (Response) : Just 73 microseconds later, at 13:34:12.344657 , the server responds to the client’s SYN packet with a SYN-ACK packet. This packet is sent from port 80 of the server to the client's ephemeral port 26232. The server acknowledges the client's request by incrementing the client's sequence number to 3531231208 and introduces its own initial sequence number (2227943007).
ACK Packet (Acknowledgment) : At 13:34:12.372629 , the client sends an ACK packet back to the server to confirm receipt of the server’s SYN-ACK packet. This packet acknowledges the server's sequence number, incremented to 2227943008. This communication occurs between the same port numbers as before: the client’s port 26232 and the server’s port 80.

At this point, the client and server are synchronized and ready to exchange data.

Note that the TCP handshake occurs before any actual data exchange takes place. The handshake is completed prior to TCP dividing the data into segments and passing them down to the Network Layer for encapsulation into packets. You can confirm this by observing that the length of the handshake packets is zero, indicating that no data is being transferred during the handshake process itself.

Sequence numbers play a crucial role, helping the sender determine whether the receiver has received the data by sending acknowledgments. When the receiver acknowledges the data, it adds the length of the received data to the sender’s sequence number.

For example, if the client sends 100 bytes of data, the server adds 100 to the client’s sequence number in its acknowledgment.

During the handshake, only 1 is added to the sequence number because no actual data is being transferred – just a synchronization request. This increment of 1 is often referred to as a Ghost Byte , as it does not correspond to any real data being sent. It simply indicates that the handshake was acknowledged.

Now, let’s analyze the last four packets in which data exchange begins:

Data Transmission Initiated by the Client : In this packet, the client (IP 92.72.34.89) sends a push (P) packet with a sequence number ranging from 3531231208 to3531231285 , indicating it is sending 77 bytes of data. This packet contains an HTTP GET request.
Acknowledgment from the Server : The server responds with an acknowledgment (ACK) for the data received from the client, indicating that it has successfully received the last byte of the client's data (up to sequence number 3531231285).
Data Transmission from the Server : The server now sends its response, which is a push packet containing an HTTP status code 200 OK along with 862 bytes of data.
Final Acknowledgment from the Client : Finally, the client sends another acknowledgment, confirming that it has received the entire response from the server up to the sequence number 2227943870.

In summary, these packets show how data is exchanged after the TCP handshake. They demonstrate how the client requests data, the server responds, and how acknowledgments help keep the connection reliable.

Troubleshooting Tip

When encountering network issues, start by analyzing network traffic and focus on the transport layer within the TCP/IP model.

This approach allows for a more streamlined troubleshooting process:

If TCP is functioning correctly: Move up the TCP/IP model to inspect the application layer.
If TCP shows issues (e.g., retransmissions or out-of-order packets): Shift focus downward to diagnose the network layer, where underlying problems may be affecting traffic flow.

This method helps isolate problems efficiently by using the TCP/IP model as a guide.

Starting at the transport layer allows you to quickly determine if the issue lies higher in the stack, such as application-level misconfigurations, or lower in the network layers, where physical connectivity or routing might be at fault.

Conclusion and Final Thoughts

In conclusion, understanding how TCP works and the Three-Way Handshake is essential for grasping how data is transmitted and troubleshooting network issues.

If you found value in this reading or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Linux Server Security

Mon, 21 Oct 2024 00:00:00 GMT

As a cybersecurity enthusiast with a passion for Linux security, I've created a series of guides and tutorials focused on enhancing the security of Linux servers.

This collection is designed to help you learn about important security concepts and practices in a logical order.

I encourage you to explore each guide to deepen your understanding and effectively secure your Linux server.

🔹 Preparing Your Ubuntu Server for First Use

Prepare your Ubuntu server for first use with essential setup steps, ensuring safe management and optimal performance from the start.

Learn more!

🔹 Managing Users on Linux Server Securely

Guide to securing your Linux server by managing users, controlling access, and configuring sudo safely.

Learn more!

🔹 Securing SSH: Essential Steps for Linux Servers

Learn key steps to secure SSH on Linux servers, including important configurations and best practices for enhanced security.

Learn more!

🔹 Securing Your Linux Server with Fail2ban

Guide to installing and configuring Fail2ban to protect your server from unauthorized access and brute force attacks.

Learn more!

🔹 Setting Up a Firewall using UFW: An In-Depth Guide

Guide to securing your server with UFW, covering setup, advanced rules, and best practices for effective firewall management.

Learn more!

🔹 How to Block Invalid Packets with UFW

Learn how to enhance your server's security by blocking INVALID packets with UFW in this step-by-step tutorial.

Learn more!

🔹 Linux Server Security: Cloud Firewall Setup

Easy-to-follow guide to enhancing your Linux server security by setting up a cloud firewall.

Learn more!

🔹 Kernel Hardening: Securing Your Linux Server

Guide to improving Linux server security with kernel hardening, protecting against network attacks and information leaks.

Learn more!

🔹 How to Keep Users' Processes Private

Learn to stop users from seeing each other's processes on your Linux server.

Learn more!

🔹 Automating Security Updates on Linux Servers

Essential steps to manually update your Linux server and set up automatic security updates.

Learn more!

🔹 Kernel Live Patching for High-Availability Linux Servers

Discover how kernel live patching boosts security and uptime for high-availability Linux servers without the need for reboots.

Learn more!

🔹 Why Linux Servers Need Reboots and How to Avoid Downtime

Learn why Linux servers need reboots and how to safely reboot your production environment with minimal downtime.

Learn more!

🔹 Preventing SYN Flood Attacks on Your Linux Server

Learn how to protect your Linux server from SYN flood attacks with firewall rules, kernel tweaks, and Fail2ban.

Learn more!

🔹 How to Protect Linux Servers from Malware

Comprehensive guide to protecting your Linux server from malware using the Maldet and ClamAV combo.

Learn more!

🔹 How to Scan for Rootkits on a Linux Server

Easy-to-follow tutorial for scanning rootkits on your Linux server using Rootkit Hunter.

Learn more!

🔹 Auditing Linux Servers with Auditd

Explore how to use Auditd to monitor and audit activities on Linux servers for improved security and compliance.

Learn more!

Preparing Your Ubuntu Server for First Use

Mon, 21 Oct 2024 00:00:00 GMT

Spinning up a new Ubuntu server is exciting – but also a bit intimidating. The moment it's online, it’s visible to the internet, and that means it's vulnerable.

I remember my first time: within hours, bots were already knocking at my SSH door.

That’s why properly preparing your server from day one is so important. It’s not just about installing software – it’s about setting the stage for security, stability, and performance.

✅

A few smart tweaks early on can help your server run faster, stay safer, and avoid problems down the line.

In this guide, I’ll walk you through the exact steps I take right after deploying a fresh Ubuntu instance. Whether you're setting up a web server, mail server, or anything else, these initial steps will help you.

I’m In!

Document Everything (Seriously)

You might not realize it now, but keeping track of what you do – from the tiniest config tweak to major installs – is one of the most valuable habits you can build as a server admin.

I learned this the hard way. The first time I messed with SSH configs and accidentally locked myself out, I couldn’t remember what I had changed – or why.

Ever since then, I document everything: the exact commands I run, what I was trying to do, and the results.

Even small changes matter. Things like:

Editing sshd_config
Installing packages or services
Adjusting firewall rules
Creating users or groups

I also note the date, time , and a quick summary of my intention (e.g., "disabled password login for root").

You don’t need a fancy setup to start – just open a plain text file on your local machine and jot things down. Later, you can explore more structured tools.

Pro Tip: Good documentation isn’t just for debugging – it’s also a huge win if you ever rebuild, automate, or hand off your setup later.

Logging In as Root

The first step is getting into your server – and for that, you'll need two things:

The server’s IP address
The root password

Many providers allow you to set the root password during server creation on their dashboard or send it to you via email with the server details, as Hetzner does.

Keep in mind that the root password provided by providers like Hetzner is often temporary. Once you access your server, you'll need to change it.

👉

New to Hetzner? Use my link to get free credits!

Once you’ve got your credentials, fire up your terminal and run:

ssh root@your.server.ip

You’ll be prompted for the root password. Type it in, and you’re in!

💡

Tip: Run passwd right away to set a new root password (or update any user’s password later). It’s a quick, essential first move.

Running Updates

Once you're logged in, your server might greet you with a message like:

X packages can be updated. Y of these updates are security updates.

Don't ignore it.

Running outdated software is one of the easiest ways to expose your server to known vulnerabilities. So, before you do anything else, let’s bring your server up to date.

Begin by updating the package list on your server with the following command:

apt update

It doesn’t install anything yet – it just tells your server what’s new in the package repositories.

Now, install the updates:

apt upgrade

You’ll be prompted to confirm – just type yes and hit ENTER.

These updates aren’t just about bug fixes – they include critical security patches, performance improvements , and stability fixes that keep your server healthy from the start.

Personal Tip: Some immediately reboot after updating, but I prefer waiting until the server setup is complete to ensure all changes apply.

Adding a Non-Root User

The root user is powerful – it has full control over your server and can run any command without restriction.

But with great power comes great risk: a single typo or wrong command run as root can break your server or open security holes.

That’s why it’s best practice to create a non-root user for your daily work. This user will have limited permissions by default and will require you to prefix admin commands with sudo – which adds a protective layer by prompting for your password before critical actions.

Run this command, replacing username with your chosen username:

adduser username

You’ll be asked to create a password and fill in some optional details.

💡

Tip: Use strong passwords! A password manager can help you generate and store complex passwords safely.

To allow your new user to run admin commands, add them to the sudo group:

usermod -aG sudo username

To check that the new user is good to go, log out with the logout command and access your server again with the new user:

ssh your.new.user@your.server.ip

Try running the apt update command. It should not work directly, as you need to run the command like this: sudo apt update and then enter your password.

✅

That’s it! You’re now set up to work safely on your server without the risks of using root directly.

SSH Keys & Hardening

By default, you log into your server with a password – but there’s a safer method: SSH key authentication.

Instead of typing a password each time, you use a key pair :

The public key goes on the server.
The private key stays safely on your local machine.

When you try to connect, the server checks if you have the matching private key. If you do, you're in.

This is far more secure than using passwords, and it protects against brute-force login attempts.

Check for Existing Keys

First, check if you already have an SSH key pair on your local machine:

ls -l ~/.ssh

If the directory is empty (or only contains known_hosts), you're good to go. If you see existing keys like id_rsa or id_ed25519, consider backing them up before continuing.

Generate a New SSH Key Pair

To create a secure key pair, run:

ssh-keygen -b 4096

You’ll be prompted to choose a file path for saving the key – press ENTER to accept the default or enter a custom path to avoid overwriting any existing key.

You’ll also be asked to enter a passphrase. While optional, using a passphrase adds another layer of security and is strongly recommended.

Once done, two files will be created in your ~/.ssh directory:

id_ed25519 – your private key
id_ed25519.pub – your public key

❗

Keep the private key safe and never share it.

Copy the Public Key to the Non-Root User

If you copy the public key for the root user, only root will be able to log in with it. If you copy it for another user, only that user can use key-based login.

Since we’re using a non-root user, we’ll copy the key for that user instead.

To enable key-based access for your non-root user , copy the public key to the server using:

ssh-copy-id -i ~/.ssh/id_ed25519.pub your.non.root.user@your.server.ip

Replace your.non.root.user and your.server.ip with your actual values. Enter the user’s password when prompted.

After this, you’ll be able to log in without a password – your key will handle the authentication.

Disable Password and Root Logins

Now that key-based authentication is working, let’s lock things down even further.

We’ll disable password logins entirely and prevent root from logging in over SSH.

Open the SSH configuration file:

sudo vim /etc/ssh/sshd_config

Find and update the following lines. If they’re commented out (start with #), remove the # character:

PasswordAuthentication no PermitRootLogin no

Once you’ve saved and exited the file, reload the SSH service to apply the changes:

sudo systemctl reload ssh

⚠️

Heads up: Some server providers include additional SSH configuration files in /etc/ssh/sshd_config.d/. Be sure to check them so your settings aren’t overridden.

Test Your Setup

Try logging in as root:

ssh root@your.server.ip

You should see:

Permission denied (publickey).

Also, try logging in as your non-root user from a machine that doesn’t have the private key – you’ll be blocked there too.

At this point, your server:

Rejects password-based logins
Blocks root SSH access
Only allows access for your non-root user with SSH key authentication

This is a major win for your server’s security.

Changing Server Hostname

Giving your server a meaningful hostname is like putting a name tag on it – it helps you recognize it at a glance and makes your terminal sessions less error-prone.

Imagine managing multiple servers and accidentally running a destructive command on the wrong one just because they all look the same in the prompt. A clear, unique hostname minimizes that risk.

But beyond convenience, setting the hostname properly is also important for functionality:

Some software relies on the server’s hostname to work correctly – especially services that depend on networking or domain resolution.
If a server's hostname can't be resolved to an IP address, it can cause communication and networking issues. This may result in timeouts, connection errors, and other unexpected behavior.

To set the hostname, use this command:

sudo hostnamectl set-hostname yourservername.yourdomain.com

Replace yourservername.yourdomain.com with the actual name you want to use. A typical hostname includes two parts:

Server name , like web or mail
Domain name , like example.com

For instance:

sudo hostnamectl set-hostname mail.example.com

That’s all it takes to update the server’s hostname.

❗

Make sure to add an A record for your hostname.

Setting Up a Firewall

Before adding services or hosting applications, it’s important to lock down your server to prevent unauthorized access. A basic firewall configuration that only allows SSH traffic is a great starting point for a secure setup.

Ubuntu comes with UFW (Uncomplicated Firewall), a user-friendly tool for managing firewall rules. On many cloud providers (like Vultr), it’s already installed – and sometimes even pre-configured to allow SSH.

UFW is typically included in Debian-based distributions, like Ubuntu, but if needed, install it with:

sudo apt install ufw

To check if UFW is already active:

sudo ufw status

If it’s inactive, continue to the next step. If it’s active and you want a clean slate, reset it:

sudo ufw disable sudo ufw reset

💡

Resetting clears existing rules, which is useful if you’re unsure what’s already in place.

Before enabling UFW, make sure SSH (port 22) is allowed – otherwise, you could lock yourself out:

sudo ufw allow 22/tcp

Now that SSH access is allowed, turn on the firewall:

sudo ufw enable

You’ll see a warning that the connection may be disrupted – but since SSH is allowed, you’re safe to continue.

✅

You now have a basic firewall in place.

Only SSH traffic is allowed in, and all other incoming connections are blocked by default. Outbound traffic remains unrestricted, so your server can still fetch updates and reach the outside world.

When you later install a web server or other services, you can open additional ports as needed:

sudo ufw allow 80/tcp sudo ufw allow 443/tcp

These commands allow traffic on the standard ports used for websites – port 80 for unencrypted web traffic and port 443 for secure HTTPS connections. Without opening these, your website won’t be accessible from a browser.

Changing the Timezone

Setting the correct timezone for your server is important because it ensures that all timestamps and scheduled tasks reflect your local time (or the time zone relevant to your server’s purpose).

An incorrect timezone can lead to confusing logs, misfired cron jobs, or mismatched timestamps in applications. Getting it right from the beginning saves you from future debugging headaches.

To see the full list of available timezones, run:

timedatectl list-timezones

This will display all supported timezones (you can scroll or pipe it through less for easier browsing).

Once you’ve found the correct timezone for your location (e.g. Europe/Berlin, America/New_York, Asia/Tokyo), apply it using:

sudo timedatectl set-timezone Your/Timezone

For example:

sudo timedatectl set-timezone Europe/Berlin

Your server’s clock is now aligned with the correct timezone.

Changing the Default Editor

Your server uses a default text editor for system-level tasks like editing crontabs (crontab -e) or secure files with visudo. By default, this editor is often Nano – a simple, beginner-friendly choice.

But if you prefer something more powerful, like Vim , you can easily switch it.

Run the following command:

sudo update-alternatives --config editor

You’ll see a list of installed editors:

There are 4 choices for the alternative editor (providing /usr/bin/editor). Selection Path Priority Status \------------------------------------------------------------ * 0 /bin/nano 40 auto mode 1 /bin/ed -100 manual mode 2 /bin/nano 40 manual mode 3 /usr/bin/vim.basic 30 manual mode 4 /usr/bin/vim.tiny 15 manual mode Press to keep the current choice[*], or type selection number:

Enter the number next to your preferred editor and press ENTER.

🙆‍♂️

I personally use Vim because of its speed and power – but go with whatever makes you comfortable!

Installing Essential Software

Once your server is updated and secure, it’s a good idea to install some essential tools that make daily tasks easier and more efficient.

Depending on what your server is for (web hosting, backups, development, etc.), you might need different tools – but some are useful in almost every setup.

Here’s a go-to list I personally install on all my servers:

sudo apt install git htop curl wget zip unzip net-tools ncdu rsync

Let’s quickly break down what these do:

git – Version control for managing code and config files
htop – An interactive process viewer (better than top)
curl, wget – For downloading files and testing web endpoints
zip, unzip – For compressing and extracting archives
net-tools – Includes legacy tools like ifconfig and netstat
ncdu – Disk usage analyzer (great for cleaning up space)
rsync – Fast, efficient file transfer and sync utility

These are lightweight but powerful tools that make server life a lot smoother. Feel free to expand this list with tools specific to your use case!

Configuring Swap Space

If your server runs out of RAM, it can crash processes or become unresponsive. One way to prevent this is by adding swap space – a portion of disk that acts as backup memory.

The caveat is that it is slower than RAM since data is written to the disk. However, it provides valuable safety against out-of-memory situations.

Personal Tip: I usually configure swap space on servers with SSD or NVMe storage. On older machines, adding more RAM is often the better choice.

First, check if your server already has existing swap space by running:

sudo swapon --show

If you receive no output, it means that swap space has not been added. You can confirm this by using the free -h command.

A common rule of thumb is to allocate swap space equal to your RAM, or double it if you have disk space to spare.

Here's how to create a 1 GB swap file:

sudo fallocate -l 1G /swapfile

Swap files must only be accessible to root. Set the correct permissions:

sudo chmod 600 /swapfile

Next, mark the file for swap usage:

sudo mkswap /swapfile

Then enable it:

sudo swapon /swapfile

Use the free -h command again to check that the swap space is now active.

To keep the swap space active after reboot, add it to the /etc/fstab file:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Your server now has a safety net when memory runs low.

Configuring SMTP Relay

To ensure your server can send important notifications – such as security alerts or upgrade failures – it’s crucial to install Postfix and configure it to use an external SMTP relay.

By relaying email through a trusted SMTP provider, you avoid common pitfalls like blocked ports or blacklisted IPs that can prevent your server from delivering mail reliably.

These notifications are especially valuable for things like:

Failed auto-updates (e.g. from unattended-upgrades)
Cron job errors
System alerts

Without a working mail configuration, you might miss out on critical issues until it's too late.

👉

I’ve written a dedicated guide on configuring Postfix with SMTP2GO, a reliable free SMTP relay that works great with most server setups.

If you're looking for full control over email delivery and want to manage your own SMTP infrastructure, I also have a detailed guide on building your own SMTP server with Postal. You can use Postal as a smart host , letting other servers send email through it securely.

Choose what fits your setup – whether it’s a third-party SMTP provider or a self-hosted solution – but don’t skip this step.

❗

A silent server is a dangerous one.

Applying Changes

Once you've gone through all the setup steps – including updates, user configuration, SSH hardening, firewall rules, swap space, and more – it’s a good idea to reboot your server.

This ensures that all changes are properly applied and any pending updates or configurations take full effect.

Run this command to reboot your server:

sudo reboot

After a few moments, your server will come back online with all configurations in place.

✅

You're now ready to move forward with whatever you’ve planned for your new server.

Planning to Use Docker?

If you're anything like me, Docker is probably one of the first tools you’ll want to install next. I use Docker on almost every server I manage, so I’ve made it a default part of my preparation routine.

Docker lets you run applications inside containers – lightweight, portable, and consistent environments that behave the same across development and production.

To get it installed properly, I recommend using Docker’s official repository rather than the default Ubuntu packages – that way, you get the latest stable releases and important security updates.

👉

I’ve written a dedicated guide that walks you through the entire process: Installing Docker and Docker Compose on Ubuntu

If Docker is part of your workflow – as it is in mine – this guide will help you set it up the right way.

Using the SSH Config File

When managing multiple servers, typing long SSH commands can get repetitive. That’s where the SSH config file comes in – it acts like an address book for your servers, making SSH connections easier and more organized.

Instead of typing the full SSH command every time, you can define shortcuts for each server.

Start by creating the config file on your local machine (not on the server):

touch ~/.ssh/config

Now, open the file with your preferred editor and add a block like this:

Host server1 HostName 81.41.156.93 User ivan Port 22 IdentityFile ~/.ssh/id_rsa

In this example:

server1 is a nickname for the server.
HostName is the server’s IP address.
User is the user you use to log in (in this case, ivan).
Port is the SSH port (22 is default, so you can skip it unless it’s custom).
IdentityFile is the path to your private SSH key.

Without a config file, connecting looks like this:

ssh -p 22 -i ~/.ssh/id_rsa ivan@81.41.156.93

But with the config file set up, all you need is:

ssh server1

Simple, right?

Now, update the values to match your own server details. If you haven’t changed the SSH port, you can skip the Port line entirely.

💡

Tip: Managing multiple servers? Just add a new Host block for each one inside the same config file.

Conclusion and Final Thoughts

🎉 Congratulations – you’ve reached the end of this guide!

By now, you’ve successfully prepared your Ubuntu server for its first use and laid a solid foundation for managing it safely and efficiently.

👉

It’s time to take the security of your server up a notch. You can find my full collection of detailed Linux server security guides here.

💬 Found this guide helpful?

I’d love to hear your thoughts, questions, or suggestions in the discussion section below. Your feedback helps improve future guides and supports others on their server journey.

Prefer a more direct conversation? Feel free to contact me anytime.

How to Install Releem on Enhance Control Panel

Fri, 31 May 2024 00:00:00 GMT

Releem is a performance tuning tool for MySQL/MariaDB that recommends changes to your database configuration based on your current workload.

It collects metrics and suggests adjustments accordingly.

Releem offers both a free plan and a premium plan.

The free plan covers 10 essential MySQL/MariaDB variables and allows you to monitor the health status of the system, the InnoDB engine, memory, and queries through their dashboard.

You can also track system metrics and MySQL/MariaDB metrics such as latency, queries per second, slow queries, and IOPS r/w.

This tutorial will show you how to install Releem on the Enhance control panel.

Preparation

First, sign up for a Releem account.

After creating your account, click the Add new server button and choose the MySQL on Linux Server: Manual Installation in Docker option.

Next, you need to add a read-only user for Releem to collect metrics.

Access the server where the database role is installed using Enhance as root.

If you are using a sudo user, switch to the root user and run the mysql command to enter the MySQL CLI.

Run the following commands:

CREATE USER 'releem'@'%' identified by '[Password]'; GRANT PROCESS, REPLICATION CLIENT, SHOW VIEW ON *.* TO 'releem'@'%'; GRANT SELECT ON performance_schema.events_statements_summary_by_digest TO 'releem'@'%'; FLUSH PRIVILEGES;

Replace [Password] in the first command with a strong, complex password.

These commands create a new user named “releem” with the specified password, grant the necessary privileges, and reload the grant tables to apply the changes immediately.

💡

The % in the previous commands acts as a wildcard, granting the “releem” user access from any host. This is important because Docker may change IP addresses. By providing global access, we prevent any issues that could arise from these IP changes.

Exit the MySQL CLI by running the exit command.

Before proceeding with the installation, we need to enable the performance schema and the slow query log.

To do this, access your Enhance panel, go to the Servers tab, and choose the server with the database role.

Under the Roles section, select the Database role and click the my.cnf button.

Then click the Edit configuration button and add the following lines to the end of the file:

performance_schema=1 slow_query_log=1

Save the changes. Enhance will prompt you to restart the MySQL container; please do so.

Now you are ready to proceed with the installation.

Installation

Now, we can install the Releem agent on the server where the database role is installed.

After selecting the MySQL on Linux Server: Manual Installation in Docker option, you’ll see an example Docker command provided by Releem for installing the agent.

However, this command won’t work directly with Enhance; we need to make some modifications.

From the provided Docker command, you need to extract your API key and the agent version.

The API key is found in the RELEEM_API_KEY variable, and the agent version is at the end of the command. As of the time of writing this tutorial, the latest agent version is 1.16.0.

Now, access the server where the database role is installed using Enhance as root. If you are using a sudo user, switch to the root user.

We need to use the following command to install the Releem agent:

docker run -d -ti --restart unless-stopped --network=enhance-network --name 'releem-agent' -e DB_HOST="[server_public_ip]" -e DB_PORT="3306" -e DB_PASSWORD="[password]" -e DB_USER="releem" -e RELEEM_API_KEY="[your_api_key]" -e MEMORY_LIMIT=1000 -e RELEEM_HOSTNAME="[server_hostname]" -v /etc/mysql/releem.conf.d:/etc/mysql/releem.conf.d -v /opt/releem/conf:/opt/releem/conf releem/releem-agent:1.16.0

Replace [server_public_ip] with your server’s public IP address, [password] with the password you set for the “releem” user, and [your_api_key] with your API key.

Set the MEMORY_LIMIT to the amount of memory you want to allocate for the database in megabytes. This means that Releem will tune the database to use this amount of memory instead of the total RAM.

If you are running both the App and Database roles on the same server, it is not recommended to allocate the total RAM to the database.

A good rule of thumb is to allocate around 60% of the available RAM.

Replace [server_hostname] with your server’s hostname.

Finally, ensure the agent version at the end of the command matches the current version.

Now run the command.

After the command finishes, you should receive an email from Releem confirming that the server is now connected.

❗

Although Releem can apply changes automatically if desired, you should manually apply the recommended configurations due to the nature of Enhance.

Conclusion and Final Thoughts

I hope this tutorial has been helpful and that everything worked smoothly for you!

In this tutorial, you’ve learned how install Releem on Enhance control panel.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Linux Server Security: Cloud Firewall Setup

Thu, 21 Mar 2024 00:00:00 GMT

Most server providers offer the option to set up a cloud firewall for your servers, allowing you to create a more advanced firewall setup.

With a cloud firewall, you’ll have both an OS-level firewall and a cloud-level firewall, effectively employing two firewalls working together.

This guide will discuss the benefits of using a cloud firewall and how to set one up, along with providing some tips about best practices.

Benefits

An advantage of having a configured and enabled cloud firewall is that if you accidentally add a rule that might lock you out from your server, you won’t be stranded.

The cloud firewall is accessible through the provider’s dashboard, allowing you to make changes as needed.

In my guide about setting up a firewall using UFW, I discussed the challenge of restricting the SSH port to our home network IP, which can change due to DHCP.

That’s why I prefer using the cloud firewall for rules that might change in the future.

Here’s my approach: I configure the UFW firewall with no specific restrictions. For example, for SSH, I allow SSH traffic from all sources on the UFW firewall. However, on the cloud firewall, I specifically restrict SSH to my IP. This way, whenever my IP changes, I can access the provider’s dashboard, update the IP, and regain access to the server.

Another significant advantage is the ability to link the cloud firewall with multiple servers.

This means that configuring and enabling the cloud firewall simultaneously applies to multiple servers, providing more convenience than changing the same rules individually.

You would make the change once and push it to all linked servers.

Returning to my approach, if my IP changes, and I want to update it in the firewall to regain access, I do it once, and I regain access to all servers connected to the same firewall.

Now, let’s discuss the default policy of cloud firewalls.

Default Policy

Most cloud firewalls have a default policy of blocking all incoming traffic and allowing all outgoing traffic, similar to the UFW firewall.

Hetzner follows the same policy, and it is the server provider I use for all my projects.

If you are also using Hetzner, you are good to go.

If not, make sure that the policy aligns with your OS-level firewall.

If the default policy is different and you are unable to change it, please contact support or refer to the documentation of your provider for further information.

However, it’s worth noting that almost all server providers follow the same policy.

👉

New to Hetzner? Use my link to get free credits!

Configuration & Activation

I will guide you through configuring and activating the cloud firewall with Hetzner.

If you are using another provider, the process should be similar.

Inside your project, go to the Firewalls tab, and click on the Create Firewall button. A new page will open where you can configure your firewall.

Add a description for each rule you add to ensure easy understanding of your configurations.

For the first rule, allowing traffic to port 22 (SSH), remove Any IPv4 and Any IPv6 , and add only your current IP.

If your IP changes, you can always access the firewall from your dashboard and update it to the new one without any issues.

Now, for the second rule, which is allowing traffic to ICMP, it’s worth noting that many network administrators consider ICMP a security risk and opt to block it at the firewall.

While ICMP does have some security issues, it also serves important functions.

Some features are useful for troubleshooting, while others are essential for certain software to function correctly.

If you have configured a firewall using UFW on the OS level, it’s important to note that UFW has the ICMP protocol open by default. I prefer leaving it this way while controlling the ICMP protocol from the cloud firewall.

I have never experienced any issues denying traffic to ICMP, but your experience may vary. I recommend trying to deny traffic first. If a problem occurs, allow traffic again to ICMP.

If you want to deny traffic to ICMP, simply remove the second rule. If you want to allow traffic again, add the rule back.

Now, add the remaining rules according to your preferences, ensuring they align with your OS-level firewall ruleset to avoid conflicts.

I always leave the Outbound rules empty as there is nothing to add.

Once you’ve completed adding the rules, scroll down to the Apply to section, and select the servers to which you want this firewall to apply.

Scroll to the end, add a name to the firewall, and click on the Create Firewall button.

Now, you have two firewalls enabled and working together.

Conclusion and Final Thoughts

Great job reaching the end!

In this guide, you’ve learned how to set up a cloud firewall.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Setting Up a Firewall using UFW: An In-Depth Guide

Wed, 20 Mar 2024 00:00:00 GMT

Setting up a firewall is essential for securing your server, and UFW makes this process straightforward and user-friendly.

Think of it as your server's bouncer – it filters network traffic and only allows authorized access, keeping unwanted traffic out.

In this guide, I’ll walk you through setting up a firewall with UFW, as well as introduce you to advanced firewall rules that allow you to take your server's security to the next level.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Author's Note

Before we dive into the details, I’d like to highlight a few points.

Most guides online only cover the basics of UFW and how to add simple user-defined rules from the command line. It's hard to find a guide that goes beyond that.

This guide will take you beyond the basics and provide an example of how to implement advanced firewall rules. I'll explain some of the configuration and rule files, and where certain settings come from.

Understanding how UFW works, how to manage rules, and the order in which they take effect is essential, and I’ll do my best to make that clear in this guide.

If there’s anything you'd like me to cover, feel free to contact me, and I’ll either update this guide or publish a new one with the information you need, or help you if you encounter any issues.

I’m also in the process of writing a dedicated guide on Linux firewalls, focusing on the inner workings of UFW and how to develop advanced solutions with it.

On Debian-based distributions, like Ubuntu, UFW often comes pre-packaged.

You can check and install it using this command:

sudo apt install ufw

Many server providers configure the UFW firewall upon deploying the server to allow only SSH connections, enabling you to connect to the server.

If you have a server from Vultr, UFW is likely to be enabled by default. In my case with Hetzner, UFW is not enabled.

👉

New to Hetzner? Use my link to get free credits!

You can check the status of UFW and your current ruleset using this command:

sudo ufw status

The command’s output will either indicate that UFW is inactive, or that it is active with your current rule set.

If UFW is currently inactive, that’s fine, as we’ll proceed to configure it properly and enable it.

However, if UFW is already active, disable and reset it using the following commands:

sudo ufw disable sudo ufw reset

You can re-enable it once you have added all the rules and finished configuring it.

UFW’s Default Policy

By default, UFW takes a secure approach by blocking all incoming traffic while allowing outgoing traffic from our server. This means our server can communicate externally, but it remains inaccessible to others.

Since there is no issue with our server reaching the outside world, there is no need to make any changes to that aspect.

However, to enable incoming traffic, it’s essential to selectively open only the required ports and authorize traffic through them.

You can find the default policy defined in the /etc/default/ufw file:

DEFAULT_INPUT_POLICY="DROP" DEFAULT_OUTPUT_POLICY="ACCEPT"

As you can see, the default policy for incoming traffic is set to DROP, while the default policy for outgoing traffic is set to ACCEPT.

If UFW is enabled, you can also review the default policy using the following command:

sudo ufw status verbose

You can modify this default behavior of UFW either by directly editing the file or by using these two commands:

sudo ufw default incoming sudo ufw default outgoing

Replace <policy> with either deny, allow or reject.

deny corresponds to DROP, allow corresponds to ACCEPT, and reject corresponds to REJECT.

Both DROP and REJECT policies prevent traffic from passing through the firewall, but they differ in their response messages.

With DROP, the traffic is silently discarded without any acknowledgment sent to the source. It neither forwards the packet nor responds to it.

On the other hand, REJECT sends an error message back to the source, signaling a connection failure.

UFW's Configuration Files

There are three files I’d like you to review.

While you may not need to modify them when configuring UFW and adding your rules, it's helpful to be familiar with them.

You may have already opened the /etc/default/ufw file and examined the default policy of UFW, but there are other settings you might need to know about.

For example, you can disable IPv6 completely by changing the IPV6 variable from yes to no.

Additionally, you have the option to modify the default forward policy and the default application policy.

You can also enable UFW to manage the built-in chains by setting the MANAGE_BUILTINS variable to yes. Built-in chains are the default chains provided by Iptables, such as INPUT, OUTPUT, and FORWARD.

UFW creates its own chains to manage its rules, such as ufw-user-input and ufw-user-output. These UFW chains are linked to the built-in chains to apply UFW’s firewall rules.

When MANAGE_BUILTINS is set to yes, on stopping or reloading UFW, it will flush the built-in chains completely, removing all rules (both UFW-managed and non-UFW rules) from INPUT, OUTPUT, and FORWARD.

Any non-UFW rules in the built-in chains will be lost. UFW will take full control of the firewall, which may conflict with other tools that manage these chains.

For this reason, I don’t recommend enabling it unless you are fully aware of the implications.

The only change I make in this file is to the IPT_SYSCTL variable. There’s another file I’ll discuss next, which is /etc/ufw/sysctl.conf. UFW uses this file to tweak certain kernel parameters.

However, the original file for changing kernel parameters is /etc/sysctl.conf.

While UFW uses its own version for this purpose, I prefer not to do that. When I harden the Linux kernel, I make my changes to kernel parameters directly in the /etc/sysctl.conf file.

If UFW modifies the same parameters in its own file, my changes won’t take effect because UFW will override any corresponding parameters in the original sysctl.conf file.

That’s why I configure UFW to use the original sysctl.conf file like this:

IPT_SYSCTL=/etc/sysctl.conf

This way, I avoid dealing with two files and can maintain a clearer overview of the changes in a single file.

If you open the /etc/ufw/sysctl.conf file, you’ll find that UFW has tweaked several kernel parameters. Once UFW is enabled, the new values for these parameters will take effect.

By default, UFW disables the acceptance of ICMP redirects, which helps prevent man-in-the-middle attacks. It also ignores bogus (invalid) ICMP errors and disables the logging of Martian packets.

Since I prefer not to manage kernel parameters in two places, I configure UFW to use the /etc/sysctl.conf file instead.

To incorporate the default changes that UFW makes, I can simply add these to the end of the /etc/sysctl.conf file:

net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 net.ipv6.conf.default.accept_redirects = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.icmp_echo_ignore_all = 0 net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.default.log_martians = 0

Now, simply reboot the server for the changes to take effect. There’s no need to enable UFW for the changes to apply, since we are using the original sysctl.conf file.

👉

Check out my kernel hardening guide, where I list all the kernel parameters I tweak to improve the security of my Linux servers.

The last file I want you to review is the /etc/ufw/ufw.conf file, which contains just two variables:

ENABLED=no LOGLEVEL=low

The first variable controls whether UFW is enabled or disabled. There's no need to change it manually, as enabling or disabling UFW from the command line will automatically update this value.

The second variable controls the log level of UFW. It can be set to off, low, medium, high, or full, depending on how much logging detail you want.

You can also change the logging level directly from the command line using the following command:

sudo ufw logging

This will automatically update the value of the LOGLEVEL variable.

UFW's Rule Files

In the /etc/ufw/ directory, you'll find files with the .rules extension:

after6.rules after.rules before6.rules before.rules user6.rules user.rules

These files control how UFW manages incoming, outgoing, and forwarded traffic. Files with the number 6 handle IPv6 traffic, while files without it handle IPv4 traffic.

It's important to note that you should not modify the user.rules or user6.rules files directly, as any changes could be overwritten by UFW. Rules added by the user from the command line are saved to these files, which is why they are called user.rules files.

You are free to add custom rules only to the before.rules or after.rules files.

The order in which UFW processes firewall rules is as follows: before.rules first, user.rules next, and after.rules last.

For instance, if you place a rule to block HTTP traffic in before.rules, it will be processed before any rule in the user.rules file that might allow HTTP traffic.

The rules in the before.rules files take priority, meaning they are executed first, allowing you to enforce more critical security measures, such as rules to block SYN flood attacks.

There are rare cases where I've had to add custom rules to the after.rules file. While I don't think you'll need to do this, it's good to at least know it exists – just in case!

And it’s important to mention that all of UFW's rule files primarily use the Iptables syntax.

Understanding the order in which UFW processes these files and their respective roles is essential for effectively managing your firewall.

Basic User-Defined Rules

In the following, I'll cover the basic firewall rules that can be added from the command line.

UFW offers a set of commands for managing firewall rules directly, allowing you to quickly specify which services or ports are allowed or denied.

While these rules are designed for basic network access control and aren't intended for advanced use cases, they are perfect for setting up a firewall swiftly and efficiently.

💡

To review the rules you've added when the firewall is disabled, use the command sudo ufw show added, as sudo ufw status won’t display the rules in that case.

Remember, every rule you add from the command line is saved to the user.rules file, so feel free to check it as you add rules to understand how UFW translates the commands into the file.

Allowing and Denying Traffic

The core functionality of UFW is to allow or deny network traffic.

To allow or deny traffic for specific ports, you use the allow or deny rules, respectively.

To allow incoming traffic on port 22 (SSH):

sudo ufw allow 22

You can also specify a protocol (TCP or UDP):

sudo ufw allow 22/tcp

To deny traffic on port 80 (HTTP):

sudo ufw deny 80

If a range of ports is required, such as 5000-6000, use the following:

sudo ufw deny 5000:6000/tcp

Similarly, to deny traffic for the same range:

sudo ufw deny 5000:6000/tcp

You can also allow or deny traffic based on a service's name instead of specifying a port number. For example, to allow SSH traffic by using the service name, you can use:

sudo ufw allow ssh

UFW will then automatically determine the correct port (port 22 for SSH) and the associated protocol (TCP) to apply the rule. This approach simplifies rule management, especially when dealing with well-known services.

Application Profiles

Applications (software or services installed) can register their profiles with UFW upon installation, enabling UFW to manage them by name.

To view the available profiles, you can use the following command:

sudo ufw app list

If your server is new, you are more likely to see only the OpenSSH profile, which is the service behind SSH.

When using application profiles, there’s no need to memorize specific ports. Instead, you use the profile name.

For instance, to allow traffic on port 443 (HTTPS), you can use the following commands:

sudo ufw allow "NGINX HTTPS" sudo ufw allow "Apache Secure"

If you’re curious about the origins of these profiles, check the /etc/ufw/applications.d/ directory.

The `limit` Rule

The limit rule in UFW helps protect against brute-force attacks by restricting the number of connection attempts an IP can make in a short time.

For example, when securing SSH, this rule lets legitimate users connect but temporarily blocks any IP that makes too many failed attempts.

By default, the rule allows only 6 connections from the same IP within 30 seconds. If the limit is exceeded, the IP is blocked temporarily, which reduces the risk of brute-force attacks.

To apply this rule for SSH traffic:

sudo ufw limit 22

This command enables SSH access while protecting the server from excessive connection attempts.

Access Control by IP or Subnet

UFW lets you control access based on IP addresses or subnets, which is useful for limiting access to specific clients or denying access from specific sources.

To allow all traffic from a specific IP to any port:

sudo ufw allow from 192.168.1.100

To allow SSH traffic only from a specific IP:

sudo ufw allow from 192.168.1.100 to any port 22

To allow traffic from a subnet:

sudo ufw allow from 192.168.1.0/24

To deny traffic from a specific IP address:

sudo ufw deny from 203.0.113.50

In some cases, you might want to specify not just the IP or subnet but also the protocol (TCP or UDP).

For example, to allow only TCP traffic from an IP address to port 80, you can specify the protocol as follows:

sudo ufw allow from 192.168.1.100 to any port 80 proto tcp

By specifying protocols in your access control rules, you gain more control over the traffic flow, ensuring that only the desired traffic (TCP or UDP) is allowed from specific sources or networks.

Enabling and Checking Status

Before activating our firewall, it’s crucial to review the rules we’ve added so far to prevent any unexpected behavior.

As I mentioned earlier, if the firewall is disabled, we can't use the sudo ufw status command to view our rules.

Instead, we use the sudo ufw show added command. This command will list all the rules we have added.

Always add the rules, review them, and then proceed to enable the firewall.

To enable UFW and apply the rules you’ve configured, use the following command:

sudo ufw enable

Now, you can check the status of UFW and your current ruleset using the sudo ufw status command.

For a more detailed view:

sudo ufw status verbose

If you experience any issues, disable UFW using sudo ufw disable and review your rules again.

If you need to reset UFW to its default state (removing all rules), you can use the sudo ufw reset command.

Deleting Rules

If, for some reason, you want to delete a rule you have added, you can use the sudo ufw delete command followed by the rule itself like this:

sudo ufw delete deny from 111.111.111.111 to any port 80 proto tcp sudo ufw delete allow 80

There is another easier way to delete rules, but it requires the firewall to be enabled. This method involves using the rule number.

Once the firewall is enabled, you can use the sudo ufw status numbered command to obtain a list of your rules and their corresponding numbers, like this:

To Action From \-- ------ ---- [ 1] 22/tcp ALLOW IN Anywhere [ 2] 22/tcp (v6) ALLOW IN Anywhere (v6)

Now, to delete a rule, you can simply use the rule number:

sudo ufw delete 1

This is a much simpler method.

Best Practices

I want to share some best practices with you from my experience

The first step before enabling a firewall is to allow SSH traffic to ensure access to the server. If you enable the firewall before adding this rule, you risk losing access to your server.

I showed you how to use the allow or limit rules. Using these rules will permit any IP address to access port 22, which means our SSH port is open to everyone. This is something I avoid on a production server.

If I have a static IP from which I can access the server, I restrict the SSH port to that IP. This provides an extra layer of security and reduces the risk of unauthorized access.

Even if you’ve generated an SSH key pair, implemented key authentication, and created a non-root user, hackers could still attempt to breach your server.

If you’ve followed my guide on preparing Ubuntu servers , you should have already completed these security steps.

Only restrict the SSH port to your IP if it’s static – such as when using a VPN service or if your ISP has assigned you a static IP.

If you have a static IP, use the following command:

sudo ufw allow from proto tcp to any port 22

Now, the IP specified in the command is the only one that can access the server.

💡

When you restrict SSH access to a single IP, Fail2ban becomes irrelevant since there are no IPs to block. However, I still recommend keeping Fail2ban installed and enabled.

Another useful feature to implement is a cloud firewall. Most server providers offer a cloud firewall, which allows you to set up a basic firewall at the cloud level and apply it to your servers.

With this, you can restrict the SSH port to your IP address using the cloud firewall while leaving it open to everyone in UFW. If your IP address changes, you can easily access your provider's dashboard to update it.

Now, let's talk about HTTP and HTTPS traffic.

If you're hosting a website, you need to allow traffic on ports 80 and 443 for visitors to access it. There should be no restrictions here, as we want everyone to reach the site.

However, there are a couple of considerations. If you're using an SSL certificate (which you should) and redirecting traffic from HTTP to HTTPS, there's no need to allow traffic on port 80. While allowing traffic on both ports is generally fine, I wanted to mention this.

Additionally, if you're using a proxy service like Cloudflare, traffic will only come from their IPs. To enhance security, restrict HTTP and HTTPS traffic to only Cloudflare's IPs.

For example, use these commands:

sudo ufw allow from to any port 80 proto tcp sudo ufw allow from to any port 443 proto tcp

Repeat these commands for all Cloudflare IPs, or automate it with a bash script.

Advanced Firewall Rules

Up until now, we’ve focused on basic user-defined rules that you can easily manage from the command line.

Now, I’ll introduce the idea of advanced rules, which allow you to control traffic at a deeper level by configuring UFW's /etc/ufw/before.rules files.

These advanced rules let you filter traffic before it reaches your server’s services and before the firewall applies its standard rules.

Advanced rules are incredibly powerful and can be tailored to specific use cases, offering finer control over your network's security and performance.

And as I mentioned earlier, all of UFW's rule files primarily use the iptables syntax.

Structure of `before.rules` Files

The before.rules file begins with a declaration of the *filter table and defines several custom chains:

*filter :ufw-before-input - [0:0] :ufw-before-output - [0:0] :ufw-before-forward - [0:0] :ufw-not-local - [0:0]

:ufw-before-input: Processes incoming packets.
:ufw-before-output: Processes outgoing packets.
:ufw-before-forward: Handles packets forwarded through the server.
:ufw-not-local: Deals with packets that are not addressed to or from the local system.

It includes several default rules to handle fundamental network operations and improve security. These rules cover a variety of scenarios, such as allowing all traffic on loopback interfaces or dropping invalid packets. They are applied by default once you enable the firewall.

The file ends with the COMMIT line, signaling the completion of the rules.

The structure in before6.rules for IPv6 is nearly identical to that of before.rules for IPv4.

The main difference is that the chains in before6.rules all have the number 6 added to their names, like this:

*filter :ufw6-before-input - [0:0] :ufw6-before-output - [0:0] :ufw6-before-forward - [0:0]

However, unlike before.rules, before6.rules does not include a ufw6-not-local chain.

It includes some of the default rules that before.rules has, but it also contains additional rules that are applied specifically to IPv6 traffic.

The file also ends with the COMMIT line, signaling the completion of the rules.

Use Cases

As I mentioned earlier, these files take priority, meaning they are executed first.

This allows us to define rules that will take effect before traffic reaches anything running on the server and before it travels further through the firewall.

You can, for example, implement a solution to block SYN flood attacks by rate-limiting the number of SYN packets allowed through ports 80 and 443, and block IPs exceeding these limits. This is something that cannot be done using basic user-defined rules from the command line.

While you can use the limit rule for ports 80 and 443, it’s not ideal for a web server since the limits may not be well-suited to handle the typical traffic patterns of a web server.

Using more advanced rules in before.rules allows you to fine-tune the firewall for specific use cases like this.

👉

Check out my guide on preventing SYN flood attacks on a Linux server, where I provide a solution that combines advanced UFW rules with Fail2ban.

Example of Advanced Rules

If you examine the contents of the before.rules file, you will notice these two rules:

-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny -A ufw-before-input -m conntrack --ctstate INVALID -j DROP

These two rules are designed to log and block any invalid packets, and they are added by default by UFW to the ufw-before-input chain, which filters incoming traffic before it reaches the server, ensuring that only legitimate connections are allowed.

UFW uses the conntrack module (short for connection tracking) to monitor connections and identify those with INVALID connection states. While these rules are effective, we can make them even better.

To further enhance the security of our server, we could add two additional rules to log and block any new connections that don’t have only the SYN flag set.

Add the following two rules below the ones added by default by UFW:

-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw-logging-deny -A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Don't forget to add them to the before6.rules file as well:

-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw6-logging-deny -A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

The first rule drops any packet that’s considered INVALID by the conntrack module.
The second rule blocks TCP packets that are flagged as NEW (indicating new connection attempts) but don’t have the SYN flag set alone.

Now, reload UFW if it is already enabled:

sudo ufw reload

These additional rules further enhance the firewall’s ability to filter out potentially malicious packets and protect your server from unwanted connection attempts.

Great job reaching the end!

I hope this guide has made setting up a firewall with UFW clear and straightforward.

👉

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Securing Your Linux Server with Fail2ban

Tue, 19 Mar 2024 00:00:00 GMT

Fail2Ban is a security tool that helps protect your server from unauthorized access attempts and brute force attacks by monitoring logs for suspicious activities and blocking the IP addresses of attackers.

This will allow your server to harden itself against these access attempts without intervention from you.

In this guide, I’ll walk you through installing and configuring Fail2Ban.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Author's Note

Before we dive into the technical details, I want to highlight a few important considerations regarding Fail2ban.

Firstly, it's crucial to understand that Fail2ban should not be your sole security measure. Some may mistakenly believe it can replace firewall rules, but that is not the case.

Additionally, disabling password authentication and relying solely on SSH key authentication may lessen the need for Fail2ban.

However, I still advocate for its use as an additional layer of security.

Installation

Fail2ban is available in Ubuntu’s repositories.

To install it, use this command:

sudo apt install fail2ban

I noticed that Fail2ban disables its service upon installation on Ubuntu 22.04 due to some default settings that may cause unexpected behavior. In contrast, it is activated by default on Ubuntu 24.04.

You can verify this by using the following command:

sudo systemctl status fail2ban

That's it for the installation.

Pre-Configuration

Fail2ban's configuration files are located in the /etc/fail2ban/ directory.

If you list the contents of this directory, you will find two important configuration files: fail2ban.conf and jail.conf.

The fail2ban.conf file contains Fail2Ban's global settings, which I don't recommend modifying.

The jail.conf file contains jails, filters with actions.

We shouldn't directly modify these files, as an update may override our changes.

That's why Fail2Ban recommends creating two local copies of these configuration files for us to modify.

Use the following commands to create a local copy of these two files:

sudo cp jail.conf jail.local sudo cp fail2ban.conf fail2ban.local

Now you can safely modify Fail2Ban's configuration.

Configuration

Open the jail.local file with your preferred editor and examine its settings.

Under the [DEFAULT] section, you will find settings that apply to all services protected by Fail2Ban.

Elsewhere in the file, there are sections like the [sshd] section, which contains service-specific settings (or individual jails) that will override the defaults.

Under the [DEFAULT] section, there are some variables that you may want to modify.

bantime

The bantime variable sets the duration for which an IP will be blocked from accessing the server after failing to authenticate correctly.

By default, this is set to 10 minutes.

You can change its value as you prefer, for instance, to 60m for one hour.

findtime maxretry

The findtime and maxretry variables function together to determine the conditions under which an IP should be blocked from accessing the server.

The maxretry variable defines the number of authentication attempts an IP is allowed to make within a time period defined by findtime before being blocked.

With the default settings, Fail2Ban will block an IP that unsuccessfully attempts to access the server more than 5 times within a 10-minute interval.

#ignoreip = 127.0.0.1/8 ::1

The ignoreip variable contains a list of IP addresses, CIDR masks, or DNS hosts that Fail2ban won't block.

By default, this variable is commented out.

I strongly advise uncommenting this line and appending your own IP address to the list. This ensures that you won't inadvertently block yourself from accessing the server.

Email Alerts

Receiving email alerts is a practice I consistently advocate for in all of my guides.

If you want to receive email alerts whenever Fail2ban blocks an IP, you should adjust these two variables inside the jail.local file:

destemail sender

The destemail variable defines the email address to which the alerts should be sent.

The sender variable defines the email address from which the alerts will be sent.

But there is something to pay attention to.

The sender variable should look like this:

sender = root@example.com

Don't use your hostname. This won't work:

sender = root@host.example.com

Lastly, there is the mta variable, which specifies the mail agent that will be used to send the emails.

By default, Fail2ban uses Sendmail as its mail agent.

If you prefer to use Postfix, you can do so without changing the mta variable, as Fail2ban will automatically use it if it is installed.

💡

Your server needs to be configured to send emails, and this can be done by setting up Postfix for external SMTP relay to ensure you receive alerts.

Now, there is one more step to take in order to receive alerts. Keep reading.

Default Action

If you scroll down a bit inside the jail.local file, you'll see the action variable:

action = %(action_)s

This variable dictates the action Fail2ban should take when blocking an IP address.

The default action is to add a firewall rule that rejects traffic from the IP address, removing it after the specified bantime elapses.

Aha, the default action doesn't send alerts, which is why I asked you to keep reading.

Above the action variable, you'll find various actions you can switch between.

For example, action_mw sends an email when taking action, action_mwl sends an email and includes logging, and action_cf_mwl does all of the above, plus sends an update to the Cloudflare API associated with your account to ban the attacker there as well.

I always prefer using the action_mwl action.

If you want to use it too, your action variable should be set as follows:

action = %(action_mwl)s

Now, whenever Fail2ban blocks an IP address, you'll receive an email alert, and Fail2ban logs the block.

Individual Jails

Now, it is time to examine the service-specific sections, also known as individual jails, such as the [sshd] jail, which protects our server from unauthorized access attempts.

Each of these jails needs to be individually enabled by adding an enabled = true line under the header, along with their other settings.

By default, only the [sshd] jail is enabled, and all others are disabled.

You can verify this by opening the /etc/fail2ban/jail.d/defaults-debian.conf file, which contains the line enabled = true for the [sshd] jail.

Scroll down the jail.conf file until you find the [sshd] jail, which should look similar to this:

port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s

If you've changed the SSH port, ensure to update the value of the port variable accordingly.

You can include variables defined in the [DEFAULT] section, such as the bantime, maxretry, and findtime variables, which will only apply to this jail.

If you scroll down further, you'll find other jails that are disabled, such as the [nginx-http-auth] or [apache-auth] jails.

If you're running NGINX and want to protect a page of your site using a password, you could enable the [nginx-http-auth] jail to secure that page from unauthorized access attempts.

Starting Fail2ban

Since the [sshd] jail, which protects SSH, is enabled, we can proceed to start and enable the fail2ban service if it is disabled, depending on the version of Ubuntu you are using.

Use the following commands to start and enable Fail2ban:

sudo systemctl start fail2ban sudo systemctl enable fail2ban

You can use the fail2ban-client command to check the active jails:

sudo fail2ban-client status

Output:

Status |- Number of jail: 1 `- Jail list: sshd

To view the status and information regarding a specific jail like the sshd jail, you can use the following command:

sudo fail2ban-client status sshd

Output:

Status for the jail: sshd |- Filter | |- Currently failed: 5 | |- Total failed: 21 | `- File list: /var/log/auth.log `- Actions |- Currently banned: 1 |- Total banned: 2 `- Banned IP list: 218.92.0.29

❗

If you have disabled password authentication for SSH, you may notice zero failed attempts.

Conclusion and Final Thoughts

Great job reaching the end!

There are many more things you can do with Fail2Ban, but for now, protecting SSH is the most important.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Automating Security Updates on Linux Servers

Mon, 18 Mar 2024 00:00:00 GMT

It is crucial to regularly update your servers.

Many hacks happen because servers aren't patched with the latest security updates.

In this guide, I'll walk you through both manual updates and setting up automatic security updates for your Linux server.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Manual Updates

First, let’s learn how to update manually.

It’s just two simple commands.

Begin by updating the package list on your server with the following command:

sudo apt update

This command prompts the server to scan the server’s packages and identify those requiring updates, including security patches.

Once this is done, run the following command to update your server’s packages that need updating:

sudo apt upgrade

The server may ask for confirmation by displaying a prompt that requires a yes or no response. Make sure to type yes.

The updating process may take a while, depending on the number of updates needed.

Automatic Security Updates

While updating manually is an option, it’s easy to forget or run out of time, which is why automatic security updates ensure your servers stay patched.

This is crucial for keeping your servers secure.

Installing Unattended Upgrades

Unattended Upgrades automatically installs security updates and patches without needing our input, so we need to install the unattended-upgrades package.

Unattended Upgrades should be installed and enabled by default on Ubuntu servers. Use the following command to check and install it:

sudo apt install unattended-upgrades

Now, we need to run just one more command:

sudo dpkg-reconfigure unattended-upgrades

A pop-up window will appear, asking you if you want to automatically download and install stable updates.

Choose < Yes> and press the ENTER key.

When you do this, Unattended Upgrades changed the value from 0 to 1 in the /etc/apt/apt.conf.d/20auto-upgrades file:

APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1";

The number indicates how often Unattended Upgrades will run in days.

A value of 1 will run Unattended Upgrades every day, while a value of 0 will disable Unattended Upgrades.

Considerations

Now that we have automatic security updates in place, there are some important considerations I’d like to share with you.

Firstly, Unattended Upgrades primarily deals with security updates.

You’ll need to manually check for other updates regularly, perhaps weekly.

Also, be aware that Unattended Upgrades might automatically reboot your server for certain updates, which could be disruptive on a production server.

I recommend to manually reboot during low-traffic periods or after notifying users of downtime.

You can customize Unattended Upgrades to either disable automatic reboots or reschedule them for less disruptive times.

Lastly, while Unattended Upgrades can be set to update all packages, not just security ones, I don’t recommend this option, as some updates may break your server.

Configuration

Unattended Upgrades has its settings in a file called 50unattended-upgrades under the /etc/apt/apt.conf.d/ directory.

Open this file in your preferred editor and take a look:

Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}"; "${distro_id}:${distro_codename}-security"; "${distro_id}ESMApps:${distro_codename}-apps-security"; "${distro_id}ESM:${distro_codename}-infra-security"; // "${distro_id}:${distro_codename}-updates"; // "${distro_id}:${distro_codename}-proposed"; // "${distro_id}:${distro_codename}-backports"; };

As you can see from this block of code, Unattended Upgrades handles only security updates.

If you want it to handle non-security updates and update other installed packages, you can uncomment the ${distro_id}:${distro_codename}-updates line.

If you only want Unattended Upgrades to handle security updates, as I recommend, ensure that only security origins are allowed and that all others are commented out, like its default behavior.

You may also want to configure whether Unattended Upgrades should reboot the server if a security update requires a reboot to be applied.

You can specify a time to reboot or disable this feature completely.

To control if Unattended Upgrades should reboot your server automatically, look for the line Unattended-Upgrade::Automatic-Reboot in the configuration file.

Set this to "false" to prevent automatic reboots after updates.

If you prefer automatic reboots, change it to "true".

Additionally, you can schedule a specific time for these reboots.

For this, find the line Unattended-Upgrade::Automatic-Reboot-Time and set it to your desired time, like "04:00" for a reboot at 4 AM.

Email Alerts

There is one more thing to be aware of.

Sometimes, Unattended Upgrades fails to install a security update automatically, requiring a manual update.

You can specify an email address to which Unattended Upgrades should send an email in case this happens.

💡

Your server needs to be configured to send emails, and this can be done by setting up Postfix for external SMTP relay to ensure you receive alerts.

Scroll down the file until you find the line Unattended-Upgrade::Mail ""; and add the email address you want to send a notification to inside the two double quotation marks.

Then, scroll down a little further until you find the line Unattended-Upgrade::MailReport "on-change"; and change it from "on-change" to "only-on-error" to receive a notification only if a security update fails to be installed.

Don’t forget to uncomment these two lines. They should look like this:

Unattended-Upgrade::Mail "hello@ivansalloum.com"; Unattended-Upgrade::MailReport "only-on-error";

Once you're done configuring Unattended Upgrades, save and close the file.

Now, ensure that the mailutils package is installed on your server, as it provides the mail command used by Unattended Upgrades to send emails.

You can install it with the following command:

sudo apt install mailutils

With that, we've completed the configuration. Now, let's test our setup.

Testing Our Setup

To verify that your configuration is working correctly, you can manually trigger an update by executing the following command:

sudo unattended-upgrade -d

You can also test your setup while setting the Unattended-Upgrade::MailReport variable to "always" to verify if your server can send emails.

Conclusion and Final Thoughts

Great job reaching the end!

In this guide, you've learned how to manually update and set up automatic security updates.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Scan for Rootkits on a Linux Server

Fri, 01 Mar 2024 00:00:00 GMT

When it comes to keeping your Linux server safe, rootkits are a major concern.

They are sneaky malware that can infect your Linux server without being noticed, making them a nightmare for server administrators.

In this tutorial, I'll walk you through the steps to scan for rootkits on your Linux server.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

What are Rootkits?

Rootkits are nasty pieces of malware that can deeply hide in your server, making it sometimes impossible to detect them.

Once a rootkit is on your server, it can do bad things like taking control, stealing important information, and letting hackers in.

Rootkits could replace commands like ls or ps with their own infected versions that work normally without you noticing.

They can infect any operating system, including Linux.

Key Considerations

Rootkits can only be planted after gaining administrative access to the server.

That's why it's crucial to prevent bad actors from gaining access to your server in the first place and planting malware.

Following essential security measures such as adding a non-root user and disabling root user access, implementing a firewall, securing SSH, and more could prevent rootkits from being installed on your server without the need for detection software.

Now, the last crucial point is that there is no software that is truly effective at detecting all rootkits. Rootkit Hunter is a good option but cannot detect every type of them.

Installing Rootkit Hunter

To install Rootkit Hunter, use the following command:

sudo apt install rkhunter

Once it is installed, the first thing to do is to run this command:

sudo rkhunter --propupd

This command is used to update the local database file with the current properties of system files.

Then we need to update the rootkit signatures but before doing this we need to adjust three things in its main configuration file.

Open the /etc/rkhunter.conf file and locate these three variables:

UPDATE_MIRRORS MIRRORS_MODE WEB_CMD

Change their values to look like this:

UPDATE_MIRRORS=1 MIRRORS_MODE=0 WEB_CMD=""

Now, use the following command to update the rootkit signatures:

sudo rkhunter --update

And that's it for installing Rootkit Hunter.

Running a Scan

After successfully installing and updating Rootkit Hunter, it is time for a scan.

Use the following command to scan for rootkits:

sudo rkhunter -c

The scan will take some time and will consume server resources, so ensure you have sufficient resources available.

When running a scan using only the -c option, Rootkit Hunter will continue to prompt you to press the ENTER key to proceed with scanning.

Use the following command instead:

sudo rkhunter -c --cronjob --rwo

The --cronjob option will make the program run as a cron job, causing it to run the entire scan without asking you to press the ENTER key.

The --rwo option will instruct the program to only log warnings instead of logging everything.

You can also use the --sk option to prevent the program from prompting you to press the ENTER key.

Conclusion and Final Thoughts

Great job reaching the end!

I hope this tutorial has been helpful for you in protecting your Linux server from rootkits.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Protect Linux Servers from Malware

Thu, 29 Feb 2024 00:00:00 GMT

Many believe Linux servers are immune to malware, but that's not entirely accurate.

While infecting Linux servers itself is tough, sharing files with Windows users or hosting CMSs like WordPress can open doors to malware.

In this comprehensive guide, I'll walk you through the steps to protect your Linux server from Malware using ClamAV (Clam AntiVirus) and Maldet (Linux Malware Detect).

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Maldet & ClamAV Combo

I always prefer to provide context before delving into technical details, so I'll start by explaining what these two software are and how they combine to form a powerful combo.

ClamAV is a free, powerful and efficient open-source software as it excels in detecting, quarantining, and removing various types of malware.

It includes the freshclam utility, which ensures regular updates to the ClamAV database. This database is continuously refreshed with the latest threats as they are discovered.

Maldet is similar to ClamAV but is specifically tailored for hosting environments.

It includes a live monitoring feature that works with the Linux kernel's inotify capability, detecting changes in files and scanning them promptly.

One of its great features is automatic creation of malware detection signatures when it detects malware on intrusion detection systems at the network's edge.

Maldet updates both its malware signatures and the program itself on a daily basis.

Now, the most interesting part is why using both of them together.

The ClamAV scan engine performs better than Maldet. When Maldet sees that ClamAV is installed, it uses ClamAV's scan engine automatically, making things run smoother.

Together, Maldet monitors specified paths for malware using ClamAV's scan engine and uses both malware signature databases.

This combined approach enhances their effectiveness compared to using either alone.

Key Considerations

Before we start, it’s crucial to be aware that ClamAV and Maldet can be resource-intensive.

Make sure your server has enough CPU and memory to handle them smoothly. Not having sufficient resources could slow down your server and affect its performance.

For a smoother experience, aim for a server with at least 2 dedicated CPU cores and 8 GB of memory.

It's a good idea to test ClamAV and Maldet on a separate server first to see how much resources they need and plan accordingly.

I've had no performance issues with them on a dedicated vCPU server from Hetzner. Their CPUs are powerful, and their memory is high-speed.

👉

New to Hetzner? Use my link to get free credits!

Lastly, it's crucial to understand that these two software aren't malware cleaners.

They detect malware, can quarantine infected files, and warn you, but they don't clean the files themselves.

Maldet will attempt to clean the files, but it's not something you can always count on. You'll need to handle file cleaning yourself.

We'll discuss this in more detail later on.

Installing ClamAV

We won't interact much with ClamAV directly when using both together.

Maldet automatically uses the ClamAV scan engine, so all we need to do is install ClamAV without any additional configuration.

However, I still want to demonstrate how to use ClamAV because there might be situations where you'll need to use it alone.

So, it's useful to know more than just how to install it.

Use the following command for installation:

sudo apt install clamav

The ClamAV database is updated automatically during the installation.

This installation adds one new service: the clamav-freshclam service, which is enabled and running by default.

As I mentioned before, this service regularly checks for database updates.

🙆‍♂️

If you're not interested in learning more about ClamAV, feel free to skip ahead to the Maldet section. Since you've already installed ClamAV, you're all set.

Running a Scan

To run a scan, use the clamscan command on the directory you want to scan.

I will scan my WordPress files for any malware or viruses:

sudo clamscan -r /var/www/webapps/wordpress/public_html

Output:

\----------- SCAN SUMMARY ----------- Known viruses: 8680618 Engine version: 0.103.9 Scanned directories: 1151 Scanned files: 6323 Infected files: 0 Data scanned: 361.83 MB Data read: 145.92 MB (ratio 2.48:1) Time: 69.349 sec (1 m 9 s) Start Date: 2024:02:28 13:33:46 End Date: 2024:02:28 13:34:55

ClamAV scanned my WordPress files and found no malware.

The -r option stands for recursive scan. When you include this option in your command, ClamAV will not only scan the specified directory but also all of its subdirectories.

This is particularly useful when you want to thoroughly check an entire directory tree for malware, ensuring that no part of the directory structure is overlooked.

Resource Insights

You'll notice that when you scan with ClamAV, it uses about a one GB of memory just to get started. This happens because it loads the database into memory.

Now, rerun the scan but keep an eye on your resource usage.

After running the clamscan command, you’ll notice a short wait before the scan begins, and the memory usage on your server will increase by around one GB or more.

ClamAV needs this one GB of free memory for the database, plus additional resources for the scanning process itself.

That's why, as I mentioned earlier, I recommend having a server with sufficient resources, or only scanning when you have enough resources available.

Some people suggest stopping the clamav-freshclam service and preventing it from starting on reboot to save resources. They recommend manually updating the database only when needed for a scan.

However, I don’t completely agree with this approach.

In my experience, the clamav-freshclam service, which checks for database updates every hour, doesn’t use a lot of resources and helps keep the database updated.

While manually updating is an option and just involves running a command before scanning, it’s not necessary.

If you still choose to stop and disable the service, you can use these two commands to do so:

sudo systemctl stop clamav-freshclam.service sudo systemctl disable clamav-freshclam.service

Now, you’ll need to manually update the database each time before you scan.

Use this command for manual updates:

sudo freshclam

❗

When using ClamAV and Maldet together, you shouldn't stop and disable the clamav-freshclam service, as it's needed for the monitoring feature. We want to make sure the database is always updated.

Scanning the Root Filesystem

You may consider scanning the entire root filesystem / for malware, but there are critical aspects to consider.

If you’re using ClamAV on a production server, run a scan only if absolutely necessary.

Certain directories like /proc, /sys, /run, /dev, and /snap are special and scanning them may lead to errors. Therefore, it’s essential to exclude these directories from the scan.

Creating a custom directory for moving infected files is advisable, and this directory should also be excluded from the scan.

To begin, create a directory for quarantining infected files:

sudo mkdir /root/quarantine

I created this directory within the root home directory.

To scan the entire root filesystem while excluding special directories, use the following command:

sudo clamscan -r --log=/var/log/clamav/clamscan.log --move=/root/quarantine --exclude='^/(proc|sys|run|dev|snap)/' /

This command performs a recursive scan of the root filesystem, excluding the specified special directories.

It logs the scan results in the clamscan.log file located in the /var/log/clamav/ directory and moves any detected malware to the quarantine directory.

The final recommendation is to avoid setting up automated scans with ClamAV. It’s better to run scans manually.

This approach is due to the unpredictable nature of scan results. There’s a risk that ClamAV might mistakenly identify a normal file as malware and move it, which could break your server.

Therefore, it’s crucial to always run scans manually and carefully examine the results to maintain the integrity of your server.

I've never had to scan the entire root filesystem because the likelihood of infection is extremely low.

However, as mentioned earlier, sharing files with Windows users or hosting CMSs like WordPress can expose your server to malware.

Therefore, it's essential to monitor these areas closely, and that 's where Maldet comes into play.

Installing Maldet

After installing ClamAV, we can proceed to install Maldet and begin configuring it.

But, make sure to install it using the root user.

If you install it using sudo, the Maldet files will be owned by the user who performed the installation, rather than by the root user. Installing it as root avoids the need to locate and modify the ownership of those files later on.

Maldet isn't available in the repositories of any Linux distribution, but it's simple to install.

After accessing the server using root, use this command to download the Maldet files:

wget http://www.rfxn.com/downloads/maldetect-current.tar.gz

Extract the archive:

tar xzvf maldetect-current.tar.gz

Now, navigate into the resulting directory and execute the installer like this:

./install.sh

Once Maldet is installed, copy the README file to your home directory so you can examine it if needed. It contains the documentation for Maldet.

The installer automatically creates the symbolic link for the maldet service, and it also automatically downloads and installs the latest malware signatures.

Now, you can access the server again using your sudo user.

There's one more thing we need to install, which is the inotify-tools package. This package is necessary for the monitoring feature and is required for the maldet service to work.

Use this command to install it:

sudo apt install inotify-tools

If we check the status of the maldet service, we'll notice that it's not active by default.

We'll start it after we finish configuring Maldet.

Configuring Maldet

Open the /usr/local/maldetect/conf.maldet file with your favorite editor.

You need to configure Maldet to alert you by email whenever it detects any malware, which is highly recommended.

Find the email_alert and email_addr variables and change them accordingly:

email_alert="1" email_addr="hello@ivansalloum.com"

There's an option to ignore email alerts for malware that has been cleaned, but I recommend disabling it.

It's enabled by default, so ensure to disable it like this:

email_ignore_clean="0"

It is always better to get notified about everything happening.

💡

Your server needs to be configured to send emails, and this can be done by setting up Postfix for external SMTP relay to ensure you receive alerts.

If you scroll down further, you'll notice the autoupdate_signatures and autoupdate_version variables, which are enabled by default and prompt Maldet to check for updates every day.

Now, we want Maldet to quarantine any infected files and attempt to clean them from any malware.

To do this, locate the quarantine_hits and quarantine_clean variables, and enable them:

quarantine_hits="1" quarantine_clean="1"

There's an important consideration to keep in mind: quarantining infected files can potentially cause issues.

For instance, if you're hosting WordPress and Maldet moves an infected file to the /usr/local/maldetect/quarantine directory, your site may stop working.

Therefore, you must decide between letting Maldet quarantine and attempt to clean the infected files, and then moving them back to the WordPress directory once you're certain they're safe, or not allowing Maldet to quarantine and clean infected files, and instead immediately cleaning them yourself after receiving an alert.

It's up to you.

Finally, we need to configure Maldet to monitor files in paths we specify.

By default, Maldet scans only files within the /dev/shm/, /var/tmp/, and /tmp/ directories.

To change this, locate the following two lines:

default_monitor_mode="users" # default_monitor_mode="/usr/local/maldetect/monitor_paths"

Comment out the first line and uncomment the second line, like this:

# default_monitor_mode="users" default_monitor_mode="/usr/local/maldetect/monitor_paths"

And that's it for the main configuration file. Save and close it.

Now, the final step before starting the maldet service is to specify the paths we want to monitor.

Open the /usr/local/maldetect/monitor_paths file with your favorite editor and add the directories you want to monitor, like this:

/home /root /var/tmp /tmp

After this, it is time to start the maldet service:

sudo systemctl start maldet

If you check the Maldet logs using the sudo maldet --log command, you will see this:

maldet(33584): {mon} set inotify max_user_watches to 16384 maldet(33584): {mon} added /var/tmp to inotify monitoring array maldet(33584): {mon} added /tmp to inotify monitoring array maldet(33584): {mon} added /home to inotify monitoring array maldet(33584): {mon} added /root to inotify monitoring array maldet(33584): {mon} added /var/www to inotify monitoring array maldet(33584): {mon} starting inotify process on 5 paths, this might take awhile... maldet(33584): {mon} inotify startup successful (pid: 33716) maldet(33584): {mon} inotify monitoring log: /usr/local/maldetect/logs/inotify_log

This output is sourced from the /usr/local/maldetect/logs/event_log file.

As you can see, Maldet has added the specified paths to the monitoring array, and the monitoring feature is now enabled.

You can also check the /usr/local/maldetect/logs/inotify_log file, as it contains useful information about the changes occurring to the files.

The ClamAV Daemon Warning

If you examine the Maldet logs again, you'll notice this warning:

maldet(33584): {mon} warning clamd service not running; force-set monitor mode file scanning to every 120s

This happens because we only installed the clamav package.

If we don't install the clamav-daemon package, you will always see this warning.

Let me explain why.

When you only install the clamav package, you'll have access to the clamav-freshclam service for updates and the clamscan utility for scans.

As explained in the Resource Insights section on ClamAV, you know that ClamAV requires a certain amount of RAM to load the database before starting a scan, and it does this each time you run a scan.

However, if you install the clamav-daemon package, you'll also have access to the clamav-daemon service. When active, this service keeps the database constantly loaded in memory.

As a result, you'll notice approximately one GB of RAM consistently in use because the database remains in memory.

This setup ensures that ClamAV is always ready to run scans, as the clamav-freshclam service continuously updates the database and the clamav-daemon service keeps it loaded.

Without the clamav-daemon service detected, Maldet won't monitor specified paths in real-time. Instead, it checks them every 120 seconds.

This is because Maldet needs ClamAV to load the database into memory before it can monitor files.

Installing the clamav-daemon package resolves this issue, and Maldet will monitor files every second.

🙆‍♂️

I wanted to address this now because I encountered this problem myself, and after some investigation, I found the solution. I wanted you to see the warning beforehand so you could understand the situation.

Now, the only remaining step is to install the clamav-daemon package and ensure the service remains active.

Use this command for installation:

sudo apt install clamav-daemon

If you check the status of the clamav-daemon service, you'll notice that it's active by default.

If you run the htop command, you'll see that approximately one GB of RAM is being used because the database is loaded into memory.

Now, restart the maldet service:

sudo systemctl restart maldet

Check the logs again, and you won't see the warning anymore.

Now, Maldet is able to monitor in real-time.

Final Testing

Maldet is active and monitoring files, eliminating the need for manual scans.

Since Maldet uses the ClamAV scan engine, there's no need to directly use ClamAV.

Additionally, Maldet creates a file within the /etc/cron.daily/ directory, which conducts a daily scan of standard web directories.

This daily scan is controlled by the cron_daily_scan variable in its main configuration file, which you can disable if desired.

To view what Maldet will scan daily, open the /etc/cron.daily/maldet file and review its contents.

Now, let us test our setup by downloading a simulated virus file from the European Institute for Computer Antivirus Research (EICAR) site.

Download one or all of the EICAR test files inside a directory Maldet is monitoring:

wget https://secure.eicar.org/eicar.com wget https://secure.eicar.org/eicar.com.txt wget https://secure.eicar.org/eicar_com.zip wget https://secure.eicar.org/eicarcom2.zip

After a few seconds, you'll notice that the file is gone and has been moved to the /usr/local/maldetect/quarantine directory if you enabled Maldet to quarantine infected files.

Additionally, you should receive an alert by email.

Conclusion and Final Thoughts

Great job reaching the end!

I hope this guide has been incredibly helpful in showing you how to protect your Linux server from malware.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Kernel Hardening: Securing Your Linux Server

Fri, 23 Feb 2024 00:00:00 GMT

While the Linux kernel is fairly secure by default, there are steps we can take to make it even safer.

Kernel hardening can reduce the risk of certain network attacks and information leaks, making it harder for attackers to plan their attacks.

This guide will walk you through adjusting key kernel parameters to strengthen the security of your Linux server.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Author's Note

When you search the internet for guides on kernel hardening, you'll come across various opinions.

I will not try to claim to have the correct information. Instead, I'll share how I harden the Linux kernel on all my servers and my experience with it.

Leaving the default kernel parameters unchanged usually isn't a problem and may not cause security issues. Tweaking some parameters just enhances your server's security.

However, this might cause problems with what you run on your server if they rely on specific values. That's why I suggest testing these tweaks in a separate environment before applying them to your main server.

Lastly, I just want to mention that I've never encountered any problems with the tweaks I'll be covering on all my Linux servers.

The `/proc` Directory

Before we dive into adjusting kernel parameters, let me first explain the /proc directory so you can understand where these parameters come from.

If you look inside the /proc directory, you'll find a bunch of numbered directories, and some regular files and directories.

The numbered directories correspond to process IDs (PIDs) of running processes.

Inside a numbered directory, you'll find files and subdirectories filled with details about the running process. However, not all files within these directories are easily understandable unless you're familiar with OS programming.

Instead, we rely on commands like ps and top which extract process information from the /proc directory and present it in an easy-to-read format.

The remaining files and directories with regular names contain details about the kernel's activity. For example, the sys directory contains the kernel parameters that can be adjusted to harden the kernel.

If you enter the /proc/sys/net/ipv4/ directory, you'll find various parameters related to IPv4 networking. Each file in this directory represents a parameter and holds its value.

Now that we know the basics of the /proc directory, it's time to begin hardening the Linux kernel.

The `sysctl` Utility

The old method of changing parameter values involves echoing the new value to the parameter's file.

This approach is outdated and doesn't work with sudo, thus requiring the use of the bash -c command to enforce execution, as illustrated by the following example:

sudo bash -c "echo '1' > /proc/sys/net/ipv4/icmp_echo_ignore_all"

However, this isn't the recommended approach. Instead, we should use the sysctl utility, which offers a more modern solution.

The following command will list all parameters along with their values:

sudo sysctl -a

Additionally, you can also check a value for a certain parameter like this:

sudo sysctl net.ipv4.icmp_echo_ignore_all

You can use these two commands to verify if the new values have taken effect after you make changes.

If you want to change a parameter's value temporarily, you could use the -w option to set the new value like this:

sudo sysctl -w net.ipv4.icmp_echo_ignore_all=1

It will last until you reboot the server.

Sometimes this is useful, but for the purpose of this guide, we want changes to be permanent.

Hardening the Linux Kernel

In the following, we'll tweak kernel parameters by adding them to the end of the /etc/sysctl.conf file along with their new values, ensuring that the changes become permanent.

After adding them, simply reboot the server for the new values to take effect.

Reverse Path Filtering

A spoofing attack occurs when an attacker pretends to be someone else or a trustworthy entity.

This can occur at various levels, including the network level, such as IP address spoofing, where the attacker sends network packets with spoofed IP addresses.

Such tactics could be used for DoS attacks or to trick access controls.

These two parameters that help us prevent spoofing attacks:

net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1

They control reverse path filtering (RP filter), a security feature that helps protect against IP address spoofing. When enabled, the server checks if it can reach the source address of incoming packets. If it can’t, the packets are dropped.

By default, these parameters are set to 2, which enables them in loose mode. Setting them to 1 enforces strict mode for reverse path filtering, providing better security.

SYN Flood Protection

One form of DoS attacks involves sending a huge number of SYN packets to a server without completing the three-way handshake, also known as SYN flood attacks.

This results in our server having numerous half-open connections, consuming significant resources and preventing it from accepting new legitimate connections.

In severe cases, it can cause the server to crash, leaving it unresponsive and inaccessible.

TCP SYN cookies help mitigate SYN flood attacks by managing half-open connections without consuming too many server resources.

The following parameter enables the use of TCP SYN cookies:

net.ipv4.tcp_syncookies = 1

While it doesn’t provide much protection, I still recommend keeping it enabled.

There are two additional parameters we can add to optimize how our server handles new connections:

net.ipv4.tcp_max_syn_backlog = 4096 net.ipv4.tcp_synack_retries = 3

The first parameter controls the size of the queue for incoming SYN requests. By default, it’s set to 256 on most systems, which is generally too low. Increasing the backlog allows the server to handle more half-open connections, helping it absorb a higher volume of incoming requests.

For most systems, a range between 4096 and 16384 is recommended unless the system has substantial memory resources.

The second parameter determines how many times the server will resend the SYN-ACK packet if no ACK response is received from the client, as happens in a SYN flood attack.

By default, it’s set to 5, meaning the server waits longer and keeps resources tied up on unresponsive clients. For better SYN flood resistance, reducing this value to 2 or 3 helps free up resources more quickly by limiting the number of retry attempts.

👉

Check out my guide on preventing SYN flood attacks using firewall rules and Fail2ban.

Disable ICMP Redirects

ICMP redirects are messages sent by a router to inform the server that it should use a different, better route for reaching a specific destination.

For example, if the server sends traffic to a router, and the router knows of a more efficient path to the destination, it can send an ICMP redirect message to the server to update its routing table.

However, ICMP packets, including ICMP redirects, are extremely easy to fake, and it would be rather simple for an attacker to forge ICMP redirect packets. This makes it a potential security risk, as attackers could use fake redirects to mislead the server into sending traffic through malicious or incorrect routes.

These eight parameters will disable the sending or accepting of ICMP redirects:

net.ipv4.conf.all.accept_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv6.conf.default.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0

In most secure environments, ICMP redirects should be disabled, especially on systems not acting as routers.

Disable Source Routing

Source routing is a feature where the sender of a packet specifies the route that the packet should take through the network, rather than allowing routers to decide the best path.

This can be useful for specific use cases but can also pose significant security risks, as attackers can exploit it to send packets along a malicious or unexpected path. It can also be used to bypass security measures.

These four parameters disable the acceptance of source routes packets:

net.ipv4.conf.default.accept_source_route = 0 net.ipv6.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv6.conf.all.accept_source_route = 0

Disabling source routing is a common security practice, particularly on servers where network traffic should follow trusted paths.

Disable Packet Forwarding

Packet forwarding enables a system to transmit network packets from one network interface to another.

Unless your server is functioning as a router or VPN, packet forwarding should be disabled.

These parameters disable packet forwarding:

net.ipv4.ip_forward = 0 net.ipv4.conf.all.forwarding = 0 net.ipv6.conf.all.forwarding = 0 net.ipv4.conf.default.forwarding = 0 net.ipv6.conf.default.forwarding = 0

Disabling packet forwarding helps prevent your server from being used as a gateway for unauthorized network traffic.

Protect TCP Connections (TIME-WAIT State)

When a TCP connection is closed, it goes into a TIME-WAIT state to prevent old or duplicate packets from interfering with new connections.

This parameter helps protect against TIME-WAIT Assassination or TCP TIME-WAIT attacks:

net.ipv4.tcp_rfc1337 = 1

When enabled, the server follows a safer behavior for closing TCP connections, ensuring that connections in the TIME-WAIT state cannot be hijacked.

This prevents potential data loss or connection issues caused by attackers exploiting this vulnerability.

However, please note that the kernel documentation has confused some people about this parameter, and there is an ongoing debate about whether one should enable it or not.

Therefore, it's advisable to test this parameter in a testing environment first.

Harden the BPF JIT Compiler

Berkeley Packet Filter (BPF) is a framework in the Linux kernel used for network packet filtering, tracing, and other performance monitoring tasks.

The Just-In-Time (JIT) compiler improves the performance of BPF programs by compiling them into machine code at runtime, instead of interpreting them line-by-line.

This parameter enables additional hardening features for the BPF JIT compiler:

net.core.bpf_jit_harden = 2

When enabled, it mitigates JIT Spraying attacks by obfuscating sensitive constants in BPF programs, such as fixed values or memory addresses.

JIT Spraying is an attack where hackers manipulate the JIT compiler to inject malicious code into memory, allowing them to control machine code execution.

By obfuscating these constants, it makes them harder for attackers to predict or exploit.

Next, add this parameter:

kernel.unprivileged_bpf_disabled = 1

It prevents unauthorized users from loading and using BPF programs and restricts the BPF JIT compiler to root-only access.

Restrict Core Dumps

A core dump is a file the Linux kernel creates when a program crashes. It shows what was in the program's memory, registers, and call stack at the time of the crash.

While useful for debugging, core dumps can expose sensitive data like passwords or encryption keys.

By default, core dumps are saved in /var/lib/systemd/coredump and compressed with zstd. They’re triggered by errors like segmentation faults or illegal instructions and can be analyzed using tools like gdb.

However, core dumps are a security risk, especially for programs with elevated privileges. If an attacker gets access, they might extract sensitive information.

That’s why it’s important to disable core dumps unless absolutely needed for debugging (and honestly, I’ve never needed them).

Two parameters can help:

kernel.core_pattern = |/bin/false fs.suid_dumpable = 0

kernel.core_pattern = |/bin/false redirects any attempt to create a core dump to the false command, effectively discarding it. While it prevents core dumps globally, some privileged processes might still bypass this restriction.

Adding fs.suid_dumpable = 0 ensures that no core dumps are created for processes with elevated privileges, adding extra protection.

Disable Magic Keys

Magic Keys are special key combinations that allow you to perform debugging and system control actions, even when the server becomes unresponsive due to issues like a kernel panic.

To disable Magic Keys, add this parameter:

kernel.sysrq = 0

Disabling Magic Keys prevents potential misuse or exploitation of these debugging features.

Restrict Access to Kernel Logs

By default, any user on the server can run the dmesg command to view kernel logs, which may contain sensitive information.

To restrict access to this command, add the following parameter:

kernel.dmesg_restrict = 1

This ensures that only users with root privileges can view kernel logs.

Restrict `ptrace` Access

The ptrace() system call allows one process (tracer) to observe and control the execution of another (tracee).

This functionality can be exploited by attackers if an application is compromised, allowing them to attach to and manipulate other processes, such as SSH sessions.

To mitigate this risk, the kernel.yama.ptrace_scope parameter can be set to restrict ptrace usage. Here’s what each value means:

**0**: Classic ptrace permissions – a process can attach to any other process running under the same user.
**1**: Restricted ptrace – processes can only attach to their descendants, or specific allowed processes declared by prctl.
**2**: Admin-only attach – only processes with CAP_SYS_PTRACE privileges can use ptrace.
**3**: No attach – no process can use ptrace to attach to another process.

For full security, setting this parameter to 3 is recommended, as it completely prevents any process from using ptrace to attach to others.

For most Linux servers, this is a good precaution unless ptrace is explicitly needed for debugging or administration.

Use the following parameter to disable ptrace unless required:

kernel.yama.ptrace_scope = 3

This ensures that no process can manipulate or inspect another process.

Restrict User Namespaces

User namespaces are a kernel feature that enhances security by creating separate sandboxes for different users.

While available for unprivileged users, this feature could expose the kernel to privilege escalation.

This parameter restricts this feature to users with root privileges only:

kernel.unprivileged_userns_clone = 0

This prevents unprivileged users from creating user namespaces.

Control Swapping

Similar to core dumps, swapping involves copying parts of the memory to the disk, potentially containing sensitive information.

To reduce this risk, we can instruct the kernel to swap only when absolutely necessary by setting the following parameter:

vm.swappiness = 1

This ensures that the server will only swap data to disk when there are no other options.

File Creation Restrictions

To enhance security, we can restrict the creation of certain types of files in potentially insecure directories.

First, we prevent the creation of FIFOs (First In, First Out, also known as named pipes) and regular files in risky locations by adding the following parameters:

fs.protected_regular = 2 fs.protected_fifos = 2

These help reduce the risk of attackers exploiting these file types in insecure directories.

Next, we protect hard links and symbolic links (symlinks) by adding the following parameters:

fs.protected_hardlinks = 1 fs.protected_symlinks = 1

The first parameter ensures that users without proper access cannot create hard links to files, while the second ensures symlinks are handled safely, helping to prevent TOCTOU (time-of-check, time-of-use) race conditions.

Together, these add an extra layer of protection against certain file system vulnerabilities.

Address Space Layout Randomization (ASLR)

Address Space Layout Randomization (ASLR) is a security technique that randomizes the memory layout of key areas in a process's address space.

This makes it difficult for attackers to predict the location of critical data areas, such as buffers, heap, and stack, which are commonly targeted in memory-based exploits.

By randomly placing memory regions, ASLR makes it much harder for attackers to successfully execute memory corruption exploits. Even if an attacker can write to memory, they won’t be able to predict where to target, as the memory layout changes consistently.

To enable ASLR, add the following parameter:

kernel.randomize_va_space = 2

This ensures that the kernel randomizes the memory layout of all processes.

Conclusion and Final Thoughts

Congratulations on reaching the end!

In this guide, you've learned how to harden the Linux kernel to enhance your server's security.

👉

For more comprehensive Linux server security resources, be sure to check out the full collection of detailed guides here.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Keep Users' Processes Private

Fri, 02 Feb 2024 00:00:00 GMT

By default, any user on a Linux server can use commands like top or htop to view all running processes, including those owned by other users.

While this can be useful in certain situations, it can also pose security risks by exposing sensitive information to potential attackers.

This tutorial shows you how to restrict users from viewing processes owned by other users, thereby enhancing the security of your Linux server.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Demonstration

To demonstrate how any user could see processes running by other users, I added a new user named Elie and ran the htop command.

Even with their normal user privileges, Elie is able to see all processes running, including those owned by root and various system users.

Keeping Users' Processes Private

To keep users' processes private, you need to add the following line to the end of the /etc/fstab file:

proc /proc proc hidepid=2 0 0

Save and close the file.

💡

This solution doesn't work for RHEL 9-type distributions. Attempting it may break your server.

Then, remount /proc like this:

sudo mount -o remount proc

Now, let me switch back to Elie and run the htop command again.

As you can see, Elie is now only able to view their own running processes.

The hidepid option with a value of 2 hides information about all running processes owned by other users, including the process directories in the /proc directory.

Conclusion and Final Thoughts

Easy, isn't it?

I hope this tutorial was helpful for you in keeping your users' running processes private.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Block Invalid Packets with UFW

Tue, 30 Jan 2024 00:00:00 GMT

Hackers use tools to create TCP packets with unusual, weird flag combinations, known as Invalid Packets, capable of causing significant harm.

UFW blocks these invalid packets by default, but there are still instances where it may overlook and fail to block.

In this tutorial, I'll guide you through the most effective way to block these types of packets.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

What are Invalid Packets?

Invalid packets include any incoming connections that don’t start with the SYN flag alone.

In a standard TCP Three-Way Handshake, every new connection begins with a SYN packet.

If a TCP connection starts with a different flag or an unusual combination of flags – like those generated by port-scanning tools such as Nmap – these packets should be blocked.

How UFW Blocks Invalid Packets

If you look at the /etc/ufw/before.rules file, you'll notice these two rules:

# drop INVALID packets (logs these in loglevel medium and higher) -A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny -A ufw-before-input -m conntrack --ctstate INVALID -j DROP

These two rules are set to log and block any packets considered INVALID, and they are added by default by UFW.

UFW uses the conntrack module, short for connection tracking, to monitor connections associated with INVALID connection states.

Let me demonstrate by using Nmap to run an XMAS scan from my MacBook against a server with UFW enabled, which has a rule allowing SSH traffic:

nmap -sX 165.22.65.229

Output:

Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 11:14 UTC Nmap scan report for 165.22.65.229 Host is up (0.0016s latency). All 1000 scanned ports on 165.22.65.229 are open|filtered Nmap done: 1 IP address (1 host up) scanned in 27.22 seconds

By default, Nmap scans only the most commonly used 1,000 ports. The XMAS scan sends invalid packets containing the FIN, PSH, and URG flags.

As you can see, UFW successfully blocked this scan, resulting in Nmap being unable to determine that port 22 is open.

The output from Nmap that All 1000 scanned ports on 165.22.65.229 are open|filtered suggests that the scan was unsuccessful.

But now, let's run a Window scan, which involves a significant number of ACK packets:

nmap -sW 165.22.65.229

This should show that port 22 is open or filtered, meaning that UFW was unable to block this scan.

Blocking Invalid Packets the Right Way

To further enhance the security of our server, we could add two rules to log and block any new connections that don't have only the SYN flag set.

Add these two rules below the ones added by default by UFW:

-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw-logging-deny -A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Don't forget to add them to the before6.rules file as well:

-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw6-logging-deny -A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Now, reload UFW:

sudo ufw reload

These rules help to block invalid and potentially malicious packets:

The first rule drops any packet that’s considered INVALID by the conntrack module.
The second rule blocks TCP packets that are flagged as NEW (indicating new connection attempts) but don’t have the SYN flag set alone.

Okay, let's run the Window scan again:

nmap -sW 165.22.65.229

Output:

Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 13:12 UTC Nmap scan report for 165.22.65.229 Host is up (0.0017s latency). All 1000 scanned ports on 165.22.65.229 are filtered Nmap done: 1 IP address (1 host up) scanned in 21.08 seconds

Great, UFW has effectively blocked this scan, preventing Nmap from determining that port 22 is open.

The `PREROUTING` Chain

There is one more thing we can do to enhance the way UFW blocks invalid packets, which is using the PREROUTING chain instead of the INPUT chain.

By leaving everything as is now, invalid packets will travel through the entire INPUT chain until they reach our rules and get blocked.

However, with the PREROUTING chain, they get blocked before reaching the OUTPUT, FORWARD, or INPUT chains. This way, we are optimizing the performance of our firewall.

Since the filter table doesn't have the PREROUTING chain, we need to use the PREROUTING chain of the mangle table instead.

To do this, add the following lines to the end of the before.rules and before6.rules files, right after the last COMMIT line:

*mangle :PREROUTING ACCEPT [0:0] -A PREROUTING -m conntrack --ctstate INVALID -j DROP -A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP COMMIT

Reload UFW:

sudo ufw reload

The mangle table is specifically used for altering packet characteristics or applying special filtering, so it’s ideal for dropping packets that don’t meet the normal connection requirements.

The PREROUTING chain is one of the first chains that incoming packets go through as soon as they reach the server. By applying our rules here, we can block invalid packets immediately, before they reach other parts of the firewall.

This reduces the load on our firewall and helps ensure that only valid traffic is processed further.

Conclusion and Final Thoughts

Great job reaching the end!

I hope this tutorial has helped you to block invalid packets and enhance your server's security.

👉

You can find the full collection of detailed Linux server security guides here.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Get a Free VPS Server

Sun, 28 Jan 2024 00:00:00 GMT

Ever wanted to kickstart a project or do some testing without breaking the bank?

Well, today I’ll walk you through getting a free VPS server. Yep, free!

Let’s dive in!

Fact or Fiction?

For a moment, you might think it's impossible – how could a VPS server be free? Surely, no company would offer that!

But the reality is, getting a free VPS server is possible. While it's not truly free and does have a price, you can obtain it at no cost from genuine server providers.

Let me break it down.

You've probably come across ads from server providers on Google, offering credits to test their servers.

These credits allow you to start for free, with some providers offering them for up to two months – a significant period.

And that's exactly what you'll do! Many server providers are willing to give you credits to get started.

Need an example? When I studied for my LPIC-1 certificate, I didn't have a Linux machine.

So, I signed up for a server provider, used the free credits to create a VPS server, and studied for the exam. After the credits expired, I simply moved on to another provider.

And yes, it's entirely legitimate.

Getting a Free VPS Server

Now that you understand the idea behind getting a VPS server for free, let me share something with you.

If I provide you with my affiliate links from these providers, you'll receive free credits to begin with. That's exactly what I'm going to do today.

It's worth noting that not all providers offer credits out-of-the-box. Sometimes you need an invitation to get them.

By using my links, you'll help me earn a commission, and in return, I'll help you obtain a VPS server for free.

But let me clarify, I didn't create this blog just to earn money through referrals.

This method genuinely helped me obtain my LPIC-1 certificate when I couldn't afford a Linux laptop or a VPS server.

I searched the internet for people sharing their invitation links, and it worked for me.

When you use my links, you not only support me but also enable me to continue creating guides and tutorials.

It's worth noting that some providers won't reward me for inviting you unless you spend real money on their services.

However, if you use my links, it strengthens my relationship with these providers.

Now, it is time to get your free VPS server:

Hetzner – 20€
Vultr – 100$
DigitalOcean – 200$

Your support by using my links is greatly appreciated.

Managing Users on Linux Server Securely

Mon, 22 Jan 2024 00:00:00 GMT

Managing users properly is one of the most important things you can do to keep your Linux server secure.

After years of running my own servers – sometimes learning the hard way – I’ve come to appreciate how critical good user management really is. It's not just about keeping things organized – it's about reducing risk and locking down potential entry points.

🔒

Think of it like this: every user account is a potential door into your server. Your job is to make sure each door is locked unless absolutely necessary – and only the right people have the keys.

One of the biggest mistakes I made early on (and I've seen others make too) was over-relying on the root account. It’s tempting because it’s convenient, but it opens the door to serious security risks. Learning how to properly use sudo was a game-changer for me – it lets you give just the right amount of access, no more, no less.

I’ll also show you how to safely edit the sudoers file – a crucial but often intimidating step for many.

In this guide, I’ll walk you through everything I’ve learned about managing users securely on a Linux server. These are tips and techniques I now use by default, and I hope they’ll save you from some of the headaches I ran into along the way.

Potential Risks of Using Root

When I first started managing my own servers, I instinctively logged in as the root user. After all, it felt powerful – it had full control over everything. But I quickly learned how dangerous that could be.

👉

Here’s the deal: The root user has unlimited power. It can execute any command – including those capable of breaking your server if you make even a tiny mistake.

But there's an even bigger issue: hackers know about this power too.

The root user is a prime target for brute-force attacks. Leaving root enabled is like leaving your front door unlocked.

What you should do instead:

Create a regular user with sudo privileges: This lets you run admin tasks safely, and you’ll be prompted for your password each time – a small but effective security layer.
Disable direct root login altogether: If root can’t log in, hackers can’t brute-force it.
Use a unique admin username: Something that’s not obvious like "admin" or your name. It’s a small trick that adds another layer of protection.

What If You Have Multiple Admins?

This is where things can really go sideways if you're not careful.

Let’s say a few team members all share the root password. What happens if one person leaves the company? Or stores the password in plain text somewhere insecure? Now you're stuck resetting the root password and sending it around again – not ideal.

Instead, here's what I've learned works best:

Give each admin their own user account.
Use sudo to assign only the privileges they actually need.
Limit access intentionally – don’t hand out root-level permissions unless absolutely necessary.

Over the years, these practices have saved me from some painful mistakes and security scares. In the rest of this guide, I’ll show you how to set all of this up, step by step – so you can manage your users confidently and securely.

Adding Users on Linux (Clearly Explained!)

Before you can manage users, you need to know how to add them – and luckily, Linux makes this pretty straightforward.

Over the years, I’ve worked with both Debian-based and Red Hat-based servers, and while they use slightly different tools, the core idea is the same.

Let's break it down.

Using `adduser` – the Friendly Way (Ubuntu/Debian)

If you're on Ubuntu or any Debian-based server, you'll probably want to use adduser. It’s a more user-friendly wrapper around the lower-level useradd command and handles a lot of things for you.

To add a new user:

adduser username

This command will:

Create a group with the same name as the user
Set up a home directory under /home/username
Ask for a password
Prompt for optional info like full name and phone number (you can just press ENTER to skip those)

💡

Tip: Avoid common usernames like admin or user. I recommend using something unique or even a randomly generated string – it makes brute-force attacks a lot harder.

Using `useradd` – the Manual Way (CentOS/Red Hat)

The useradd command is more common on Red Hat-based systems like CentOS, but it's also available on Ubuntu. It's more barebones and gives you more control, which makes it great for scripting — but also easier to mess up if you're not careful.

While it creates a new user like the adduser command, it has a few differences:

It does not create a home directory.
It does not set a password.
It does not ask for any user details.

To make it behave more like adduser, you'll need to provide a few flags:

useradd -m -d /home/username -c "Full Name" -s /bin/bash username

Here’s what each flag does:

-m: Creates a home directory
-d /home/username: Sets the path of the home directory
-c "Full Name": Adds a comment (often used for the full name)
-s /bin/bash: Sets the default shell to Bash

Then don’t forget to set the user’s password:

passwd username

You’ll be prompted to enter and confirm the new password.

💡

Shell matters: When you use adduser, the default shell is /bin/bash. But useradd (unless you specify otherwise) sets the shell to /bin/sh, which is more limited. For most users, especially new ones, Bash is the better choice.

So, Which One Should You Use?

If you're doing things manually or managing a small number of users, adduser is almost always the better choice. It’s simple, clean, and gets everything set up with minimal effort.

That said, useradd shines when you're writing scripts or doing automated setups where you don’t want interactive prompts.

Personally, I use adduser for one-off tasks and useradd in automation. Knowing both gives you flexibility depending on the situation.

Locking Down Home Directories

One detail that often slips under the radar – especially when working with different Linux distributions – is how home directories are secured by default.

Each distro handles this a little differently, and if you're not paying attention, you could end up with user directories that are readable by everyone on the server. Not ideal.

Let’s see what happens on Ubuntu 20.04

First, let’s add a user named john:

useradd -m -d /home/john -s /bin/bash john

Then, check the permissions on the /home directory:

ls -l /home

Output:

total 4 drwxr-xr-x 2 john john 4096 Jan 19 08:50 john

See that? John's home directory has r-x (read and execute) permissions for everyone. That means other users on the server can view his files or even browse the folder. Definitely not what you want for sensitive user environments.

Setting Secure Defaults with `useradd`

Sure, users can change directory permissions manually using chmod, but there’s a better way: set secure defaults.

Open /etc/login.defs, find the UMASK line and change its value to 077 like this:

UMASK 077

This setting ensures new users will have locked-down home directories by default.

Let’s try it again. This time, I’ll create a user named ivan:

useradd -m -d /home/ivan -s /bin/bash ivan ls -l /home

Output:

total 4 drwx------ 2 ivan ivan 4096 Jan 19 09:02 ivan drwxr-xr-x 2 john john 4096 Jan 19 08:50 john

Success! Ivan’s directory is now fully private (drwx------), while John's is still open.

But Wait – What About `adduser`?

If you use adduser instead, it won’t follow the UMASK setting. Instead, it uses a config file of its own: /etc/adduser.conf.

Open that file and change the DIR_MODE value:

DIR_MODE=0700

Now adduser will also create private home directories moving forward.

💡

Heads-up for Red Hat users: Red Hat and its derivatives (like CentOS) already use secure home directory defaults out of the box – so no changes are needed there.

Ubuntu 22.04+ – Good News!

Starting with Ubuntu 22.04 and 24.04 , things have improved. Both useradd and adduser now use more secure default settings.

In /etc/login.defs you'll find:

HOME_MODE 0750

In /etc/adduser.conf:

DIR_MODE=0750

This means home directories are still private to the user and their group – a reasonable default for most setups.

Password Management

Now that you've seen how to add users, it’s time to talk about managing their passwords – securely and efficiently.

It’s not enough for each user to just “remember” their password. In any serious setup, you need a centralized, secure way to store and share credentials – and that’s where password managers come in.

These days, using a password manager isn’t optional – it’s essential.

Personally, I use and recommend Proton Pass. It’s fast, secure, and works seamlessly across devices. Their Business plan supports team use, making it easy to securely share passwords and sensitive credentials within your organization – without sacrificing performance or privacy.

👉

Bonus: If you sign up using my link, you’ll get 50% off your first year.

Having one secure vault for storing and generating strong passwords saves you from reusing weak ones – and makes enforcing strict password policies a lot more practical.

Enforcing Strong Passwords

Once you’ve got a manager in place, the next step is making sure users create strong passwords in the first place.

To do this, we’ll use the pwquality module, part of PAM (Pluggable Authentication Modules).

Let's begin by installing the module:

apt install libpam-pwquality

Next, open the config file:

vim /etc/security/pwquality.conf

This file is well-documented, so feel free to explore. But to get started, focus on these two important settings:

minlen = 32 minclass = 4

Here’s what they mean:

minlen: sets the minimum password length
minclass: sets the minimum number of character types (uppercase, lowercase, digits, symbols)

Why so strict? Because users won’t have to memorize these – they’ll use the password manager to generate and store them securely. So go big on security here.

⚠️

Note: Users with root privileges can bypass these rules when setting passwords.

I’ll leave the rest up to you to meet your own criteria – as mentioned earlier, the file is well-documented. Just make sure to uncomment the variables for your changes to take effect.

After saving the file, test it by creating a new user and trying to set a weak password. You’ll see the policy in action.

Password Expiration Policy

When you add a new user, you typically assign them a password and store it securely in your password manager.

But that’s just the start – you also want to enforce a password rotation policy so users change their password regularly.

In this example, we’ll require users to change their password every 30 days.

To enforce this, open /etc/login.defs and set the PASS_MAX_DAYS variable like this:

PASS_MAX_DAYS 30

This means each user must change their password within 30 days of the last password update.

After adding a new user, you can verify their expiration policy with:

chage -l username

This command shows info like when the password was last changed and when it will expire.

💡

Note: Any user can run this command for their own account – no root privileges needed – but they can’t view or modify anyone else’s data.

Let’s test this by adding a user named elie and checking the output:

Last password change : Jan 25, 2024 Password expires : Feb 24, 2024 Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : 30 Number of days of warning before password expires : 7

As you can see, Elie’s password will expire 30 days after it was last set. He’ll also receive a warning starting 7 days before expiration – in this case, beginning on Feb 17, 2024.

Optional: Control When Passwords Can Be Changed

By default, users can change their password any time before the expiration date – even daily. While that might sound secure, it’s not a policy I recommend.

Personally, I prefer users to only change their password during the 7-day warning window. That way, they can’t rotate passwords unnecessarily – and are gently nudged to update only when needed.

To do this, we set the PASS_MIN_DAYS variable in /etc/login.defs to 23, like so:

PASS_MIN_DAYS 23

This setting ensures users must wait at least 23 days before changing their password again – which lines up perfectly with the 7-day warning period (days 24–30 after their last password change).

Implementing `sudo` the Right Way

When sudo is set up properly, it significantly boosts the security of your server.

Here’s what I love about sudo, and why I always recommend using it over the root account:

Granular control: Give some users full root access, while limiting others to specific commands only.
No root password sharing: Users authenticate with their own password – no need to distribute the root password to your whole team.
Better security: You can disable the root account , making brute-force attacks far less effective. Attackers also won't know which usernames have elevated privileges.
Accountability: Every use of sudo is logged , so you can track exactly who did what and when.

And that’s just scratching the surface – let’s look at how to implement it securely and effectively.

Granting Full Root Privileges to a User

You’ve probably seen this recommended before: create a non-root user, add them to the sudo group, and disable root login.

But have you ever wondered what actually gives that group its power?

The answer lies in the **/etc/sudoers** file – the heart of sudo configuration.

How `sudo` Permissions Work

Whenever a user tries to run a command with sudo, the server checks the /etc/sudoers file to see if they’re allowed to do so.

To safely edit that file, use the visudo command. Never edit it directly with a regular text editor , or you risk locking yourself out due to syntax errors.

If visudo opens with an editor you don’t like (e.g., nano), you can change it:

update-alternatives --config editor

Select your preferred editor and then run visudo again.

Inside the file, you’ll see something like this:

# User privilege specification root ALL=(ALL:ALL) ALL # Members of the admin group may gain root privileges %admin ALL=(ALL) ALL # Allow members of group sudo to execute any command %sudo ALL=(ALL:ALL) ALL

The % symbol means it's referencing a group. So when a user is added to the sudo group, they inherit full root-level privileges.

Let’s break down that last line:

ALL: Applies on all hosts
**(ALL:ALL): **Run commands as any user and any group
ALL: Run any command

That’s full root access – without needing to touch the root password.

Add a User to the `sudo` Group

Here’s how to add a user named ivan to the sudo group:

usermod -aG sudo ivan

Now, Ivan can run commands like this:

sudo apt update

...and enter his own password – not the root user’s.

💡

Note: On Red Hat-based servers (like CentOS or RHEL), the equivalent group is called wheel instead of sudo.

Manual `sudo` Privileges in the `sudoers` File

You don’t have to rely on groups. You can give individual users specific rules.

Let’s say you want to give full privileges to ivan. In the sudoers file, just add this line under the root entry:

ivan ALL=(ALL:ALL) ALL

This grants Ivan the same full access as root – across all users, groups, hosts, and commands.

Add a User with `sudo` From the Start

Want to create a new user and give them sudo access immediately? Use this:

useradd -G sudo -m -d /home/john -s /bin/bash ivan

Let’s break that down:

-G sudo: Adds Ivan to the sudo group
**-m: **Creates a home directory
-d /home/ivan: Sets the home directory path
-s /bin/bash: Sets the default shell to Bash

This saves a step and gets your user set up with privileges right away.

Customized User Privileges

So far, we’ve seen how to give users full root access with sudo. But most of the time, you don’t want to do that.

Instead, it’s best to follow the principle of least privilege : give each user just the access they need to do their job – and nothing more.

Let me show you how to set that up with sudo.

Scenario: Software Team Access Only

Imagine this situation: you have two users on the Software team – ivan and john. You want them to be able to install and manage packages, but not touch anything else on the server.

Sure, you could add separate entries for each of them in the sudoers file, but there’s a better, cleaner way – using aliases.

Open the sudoers file with visudo, and create a user alias for the Software team:

# User alias specification User_Alias SOFTWAREADMINS = ivan, john

This way, instead of repeating usernames, you can refer to SOFTWAREADMINS as a group.

Now, define a command alias for the allowed software-related commands:

# Cmnd alias specification Cmnd_Alias SOFTWARECOMMANDS = /usr/bin/apt, /usr/bin/dpkg

You can add more commands here if needed – just separate them with commas.

Finally, give the SOFTWAREADMINS group permission to run those commands using sudo:

SOFTWAREADMINS ALL=(ALL:ALL) SOFTWARECOMMANDS

This line says: All users inSOFTWAREADMINS can run SOFTWARECOMMANDS as any user or group, on any host.

The beauty of this setup is how easy it is to manage:

If someone joins the Software team , just add their username to the SOFTWAREADMINS alias.
If you need to allow a new command , just add it to the SOFTWARECOMMANDS alias.

This keeps your sudoers file clean, readable, and scalable – especially on servers with multiple roles or teams.

Be Careful: How You Specify Commands in `sudo` Matters

Here’s a common mistake I’ve seen (and made myself early on): defining a command in asudo alias without any arguments.

Let’s break down what that means – and why it can be risky.

The Problem With Broad Command Permissions

In our earlier example, we created this command alias:

Cmnd_Alias SOFTWARECOMMANDS = /usr/bin/apt, /usr/bin/apt

At first glance, it seems fine – we’re just giving access to package management commands, right?

But here’s the issue: when you list a command alone , users can run it with any arguments, subcommands, or options.

This means:

They can install software
But also remove , purge , or even break things unintentionally

That’s probably not what you want.

Limiting What Commands Can Actually Do

Let’s say we only want users to be able to update and install software – not remove or purge it.

You might try this:

Cmnd_Alias SOFTWARECOMMANDS = /usr/bin/apt update, /usr/bin/apt upgrade, /usr/bin/apt install

This seems safer – but try running:

sudo apt install postfix

It won’t work.

Why? Because we didn’t specify any argument for install. As far as sudo is concerned, only apt install with no arguments is allowed – and that’s useless.

To allow users to run apt install with actual package names, modify the alias like this:

Cmnd_Alias SOFTWARECOMMANDS = /usr/bin/apt update, /usr/bin/apt upgrade, /usr/bin/apt install *

The * wildcard tells sudo: Allow any arguments after this command.

Now users can do:

sudo apt install nginx sudo apt update sudo apt upgrade

But they still can’t run other unrelated or destructive commands.

Same Rule, Different Context: Managing Services

This principle applies beyond just package management.

Let’s say you're managing a services team who needs to check the status of running services. You might be tempted to write:

Cmnd_Alias SERVICECOMMANDS = /usr/bin/systemctl, /usr/sbin/service

Sounds reasonable, right?

Wrong. This would give them access to commands like:

sudo systemctl reboot sudo systemctl isolate rescue.target

That's way more access than you intended.

Restricting `systemctl` to Status Only

To allow only safe, read-only actions, like checking service status, define the alias more precisely:

Cmnd_Alias SERVICECOMMANDS = /usr/bin/systemctl status *

Now users can run:

sudo systemctl status ssh sudo systemctl status nginx

But they can’t restart, stop, or disable services – which keeps the server safe.

A More Specific Example

Let’s say you trust a user to manage only the SSH service , but allow full control over it (start, stop, restart, etc.).

Use this:

Cmnd_Alias SERVICECOMMANDS = /usr/bin/systemctl * ssh

This allows things like:

sudo systemctl restart ssh sudo systemctl status ssh

But they still can’t touch nginx, mysql, or anything else.

Best Practices for Command Aliases

Never specify a command alone (e.g., just /usr/bin/systemctl)
Always include arguments or wildcards to precisely define what's allowed
Think through what the user really needs – and limit everything else
Test your configuration with sudo -l as the user to confirm permissions

⚠️

My rule of thumb: Never leave a command bare in a sudo alias. Be specific about what comes after – it’s the only way to safely control what users can do.

Mitigating Shell Escapes for Users

Some programs – like text editors and pagers – allow users to escape into a shell and run commands without exiting the program. This feature is called a Shell Escape , and it can be a serious security risk if left unchecked.

Let me show you what that looks like.

Example: Escaping Shell From `vim`

Let’s say you're working inside vim, and you run this:

:!ls

This will list files in the current directory – like so:

bib.csv lilly.sh script.sh

Seems harmless, right? But here’s where it becomes a problem...

When Limited `sudo` Access Becomes Full Root Access

Let’s say we gave a user – ivan – permission to edit the SSH config file using vim:

ivan ALL=(ALL:ALL) /usr/bin/vim /etc/ssh/sshd_config

That should mean Ivan can only edit that one file.

But now watch what happens if Ivan runs this inside vim:

:!apt update

Or worse:

:!bash

Output:

root@testing:/home/ivan#

Just like that, Ivan is in a full root shell – with unrestricted access to the entire server. All from within vim.

Fix 1: Use `sudoedit` Instead

To avoid shell escapes entirely, use sudoedit instead of directly calling vim:

ivan ALL=(ALL:ALL) sudoedit /etc/ssh/sshd_config

With sudoedit, users still get to edit the file, but without the ability to escape into a root shell.

If Ivan tries to run :!bash, they'll land in their own unprivileged shell – not root.

Fix 2: Use the `NOEXEC` Flag

Some other programs like less, more, or emacs also support shell escapes. For these, you can use the NOEXEC option in the sudoers file to block those escape attempts.

For example:

ivan ALL=(ALL:ALL) NOEXEC: /usr/bin/less /etc/ssh/sshd_config

Now, Ivan can still view the file using less, but cannot drop into a shell from within the program.

Why Not Just Use `NOEXEC` With `vim`?

You can – and it works:

ivan ALL=(ALL:ALL) NOEXEC: /usr/bin/vim /etc/ssh/sshd_config

But in this guide, I wanted to show both methods – because sometimes using sudoedit is cleaner, especially when the user only needs to edit a file, not run a full program like vim.

Preventing Abuse of Shell Scripts

Let’s say Ivan writes a shell script that requires root privileges to run. He asks you to allow it using sudo.

You might be tempted to add this line to your sudoers file:

ivan ALL=(ALL:ALL) /home/ivan/script.sh

Now Ivan can run the script as root – but here's the problem: he owns the script.

And since he owns it, he can edit it anytime he wants. That means he could add something like:

sudo -i

...to open a full root shell from inside the script – essentially giving himself unrestricted access. That’s a serious security hole.

The Right Way to Handle User-Created Scripts

Here's the safe process I always follow:

Let the user create and test the script in their own home directory
Review the script thoroughly
Move the script to a trusted server directory, like /usr/local/sbin
Change the owner to root and restrict write access
Then – and only then – grant sudo access to that script

Now, Ivan can run the script with elevated privileges, but he can’t modify it – which prevents abuse.

Viewing Your `sudo` Privileges

Not sure what you’re allowed to do with sudo? There’s a simple way to check:

sudo -l

This command lists your current sudo privileges, along with any environment-related settings applied through sudo.

💡

Tip: Run sudo -l often, especially when troubleshooting or verifying access. It's a simple way to confirm exactly what commands you’ve been granted – and whether your sudoers entries are working as expected.

Understanding the `sudo` Timer

Ever notice how, after entering your password once with sudo, you can run more sudo commands for a while without being asked again?

That’s thanks to the sudo timer. By default, sudo remembers your authentication for 5 minutes. During that window, you can run additional sudo commands without re-entering your password.

While convenient, this can be a security risk – especially on shared servers or if a user steps away from their keyboard without locking the terminal.

To avoid this, you might want to disable the sudo timer entirely, requiring the password every time sudo is used.

How to Disable the `sudo` Timer

To make sudo always prompt for a password, add this line to the Defaults section of the sudoers file (using visudo):

Defaults timestamp_timeout=0

Now, users will be prompted for their password every time they use sudo.

If you're working on a server where the timer is still active (e.g., the default 5-minute window), and you’re about to walk away but don’t want to close the terminal, you can manually expire the session with:

sudo -k

This immediately clears the cached credentials. Any future use of sudo will require re-authentication – even if it’s within the timer window.

Monitoring User Activities with `sudo`

One of the best things about sudo – beyond security – is transparency.

Every time a user runs a command with sudo, it gets logged. This gives you a clear audit trail of who did what and when.

By default, sudo logs activity in your server’s main authentication log – usually:

/var/log/auth.log

While that works fine, I personally prefer keeping sudo logs separate. It makes things cleaner and easier to investigate when you're troubleshooting or auditing user behavior.

To direct all sudo activity to its own dedicated log file, just add this line to the Defaults section of your sudoers file (use visudo):

Defaults logfile=/var/log/sudo.log

Once set, all sudo commands – both successful and failed – will be logged to:

/var/log/sudo.log

This includes:

What command was run
When it was run
By which user
And whether it succeeded or failed

It's a small change that goes a long way in maintaining visibility on your server.

Conclusion and Final Thoughts

Awesome job making it to the end!

I hope this guide has given you a clear, practical understanding of how to securely manage users on a Linux server. If you’ve followed along and implemented these steps, you’ve already taken meaningful action to strengthen your server’s security.

👉

Looking for more? Check out my full collection of in-depth Linux server security guides.

💬 Found this guide helpful?

I'd love to hear about your experiences, questions, or ideas in the discussion section below. Your feedback not only helps improve future guides – it helps fellow admins on their own journey.

Prefer a more direct conversation? Feel free to contact me anytime.

Securing SSH: Essential Steps for Linux Servers

Wed, 10 Jan 2024 00:00:00 GMT

Last year, I woke up to dozens of failed login attempts on a client’s web server. Bots were hammering port 22 every few seconds – and if even one guess had landed, they’d have had full root access.

🙆‍♂️

That night, I realized: SSH’s defaults are convenient, but they’re also a glaring invitation to attackers.

These days, one of the first things I do on any fresh server is lock down SSH. The out-of-the-box config is functional, sure – but it leaves way more doors open than I’m comfortable with.

In a previous guide, I covered a few essential SSH tweaks for new Ubuntu setups. In this one, we’re going deeper.

We'll walk through the sshd_config file line by line, and I’ll share the exact changes I make on every server I manage.

Along the way, I’ll include practical, real-world tips – not just theory – so you can confidently harden your SSH setup without locking yourself out.

The Basics – What SSH Really Is (And Why the Server Side Matters)

SSH (Secure Shell) is a protocol that lets you securely connect to and control a remote server over a network – usually from your terminal or an SSH client.

SSH is split into two parts:

The server-side daemon (sshd) that runs on the server you want to access
The client (ssh) that you use to connect to it – usually from your own machine

When I'm managing servers, I almost always connect from my Mac’s terminal. If you’re on Windows, a tool like PuTTY**(or Windows Terminal with OpenSSH) will do the job.

But here's the important part: security decisions are made on the server side.

You can set whatever preferences you want on your SSH client – but at the end of the day, it’s the server (via the sshd_config file) that controls whether password logins are allowed, which keys are accepted, and so on.

You’ll find SSH settings in these places:

Server: /etc/ssh/sshd_config
Client-wide: /etc/ssh/ssh_config
Per-user: ~/.ssh/config

🙆‍♂️

In this guide, I’m focusing entirely on server-side SSH – the stuff that actually locks down your server.

Before You Start: A Few Things I’ve Learned the Hard Way

I've hardened dozens of servers over the years, and one thing I’ve learned is that not every setting works for every situation.

You need to think through how your server is used before blindly applying changes. A tweak that improves security in one setup might break something important in another.

For example: disabling password authentication is a great move – but if no key-based access is set up yet, you'll lock yourself out.

Here are a few best practices I personally follow every time:

Back up your config file before touching anything.
Most options in the file are commented out. Uncomment them before making changes.
Some server providers (like Hetzner) use config overrides in /etc/ssh/sshd_config.d/. Always double-check that directory if something you changed isn’t working.

👉

New to Hetzner? Use my link to get free credits!

How I Reload SSH Safely (So I Don’t Lock Myself Out)

Once you’ve edited your sshd_config file, here’s the exact process I use to apply changes without risking a disconnect.

Keep a second session open
Before you reload, open a second SSH session and leave it connected. If a config change locks you out, that live session is your way back in to fix it — without it, a single typo in sshd_config can mean a trip to the rescue console.

Test for syntax errors:

sudo sshd -t

If there's a mistake, the command will tell you – before you break SSH access.

Reload the SSH service (don ’t restart it):

sudo systemctl reload ssh

Reloading is safer than restarting. It applies the new config but keeps your existing connection alive – which has saved me more than once.

Change the Default SSH Port

SSH runs on port 22 by default – and every scanner on the internet knows it. One easy way to reduce background noise (and random login attempts) is to move SSH to a non-standard port.

Open your SSH config file:

sudo vim /etc/ssh/sshd_config

Look for the Port line – it’s usually commented out and set to 22 by default. Uncomment it and choose a different port number:

# [!file /etc/ssh/sshd_config]
#Port 22
Port 592 # [!code highlight]

Avoid using ports that are already taken by common services (like 80, 443, 3306, etc.). Pick something between 1024 and 65535 that doesn’t conflict with anything else on your server.

From now on, when you SSH into your server, you'll need to include the -p flag with your new port:

ssh -p 592 username@your.server.ip

✅

Personal Tip: I usually keep one terminal open on the old connection while testing the new port – just in case something goes wrong.

Add a UFW Rule to Protect SSH

Changing your SSH port helps reduce noise, but a firewall adds an extra layer of protection – especially against brute-force attacks.

If you're using UFW (which ships with Ubuntu), you can rate-limit SSH connections like this:

sudo ufw limit 22/tcp

The limit rule allows connections but throttles repeated attempts from the same IP – perfect for slowing down brute-force bots without locking yourself out.

⚠️

If you changed the SSH port, make sure you limit that port, not just the default 22.

You can check your firewall status with:

sudo ufw status

And if UFW isn’t enabled yet:

sudo ufw enable

🙆‍♂️

I always apply this step after changing the SSH port – it’s quick, simple, and adds another layer of defense.

Fine-Tuning SSH Login Behavior

While scrolling through your sshd_config file, you’ll likely spot the following three lines – commented out by default:

#LoginGraceTime 2m #MaxAuthTries 6 #MaxSessions 10

These settings control how users connect to your server – how long they have to log in, how many times they can try, and how many sessions they can open. The defaults are fairly relaxed, but in most cases, they’re more permissive than they need to be.

Here’s how I typically adjust them to reduce risk without getting in the way of normal usage:

LoginGraceTime 20 MaxAuthTries 3 MaxSessions 5

What these do:

LoginGraceTime 20: This shortens the authentication window from 2 minutes to just 20 seconds. If someone can’t log in within that time, it’s usually a sign something’s wrong – or someone’s probing your server.
MaxAuthTries 3: Reducing the number of allowed login attempts from 6 to 3 helps block brute-force attacks more quickly. I’ve found that 3 strikes is plenty if you're using keys or proper credentials.
MaxSessions 5: By default, users can open up to 10 concurrent sessions. That’s rarely necessary. Cutting it down to 5 adds another layer of control without affecting usability.

🙆‍♂️

These tweaks are subtle but effective , and they’re part of my default SSH hardening routine on every server I manage.

Prevent Logins with Empty Passwords

Yes, it’s technically possible on a Linux server to create users without passwords – but it’s never a good idea. That would allow anyone to log in without authentication, which is obviously a huge security risk.

Thankfully, SSH is configured to block empty-password logins by default – but I always like to double-check this setting just to be safe.

Open your sshd_config file and look for:

#PermitEmptyPasswords no

As long as it's set to no (even if it’s still commented out), you’re good. No one with an empty password will be allowed to connect over SSH.

If, for any reason, it's set to yes, change it to:

PermitEmptyPasswords no

✅

Personal Tip: I usually leave this one commented, but I always verify it's not accidentally changed – especially on servers I didn’t set up myself.

Set an Idle Timeout Interval

Leaving an SSH session open and idle – especially on a shared or unattended machine – can create a serious security risk.

To reduce that risk, I always configure SSH to automatically close idle connections after a short period of inactivity.

In your sshd_config file, these two settings control that behavior:

#ClientAliveInterval 0 #ClientAliveCountMax 3

By default, SSH does not close idle connections unless you explicitly configure it to do so. Here’s what the default values mean:

**ClientAliveInterval** is set to 0, which means idle timeout is disabled by default
**ClientAliveCountMax** is set to 3, but it only takes effect if ClientAliveInterval is greater than 0

Uncomment and update the values like this:

ClientAliveInterval 60 ClientAliveCountMax 3

Here’s what they do:

ClientAliveInterval: This sets the number of seconds the server waits before sending a keep-alive check to the client. I typically set it to 60, meaning the server checks in once per minute.
ClientAliveCountMax: This defines how many missed responses are allowed before the server disconnects the session. With a value of 3, that gives users up to 180 seconds (3 minutes) of inactivity before the session is closed.

✅

With these settings in place, idle SSH sessions will be automatically closed after 3 minutes – a great safety net without being too aggressive.

Disable Rhosts-Based Authentication

Rhosts-based authentication is an old, insecure method that relies on trusting users based on their IP address or hostname – without requiring a password or key. It was more common in the early days of Unix systems, but it’s now considered unsafe and outdated.

SSH disables it by default , but I always recommend double-checking – especially on older servers or images that may carry legacy settings.

In your sshd_config file, look for:

#IgnoreRhosts yes

If it’s already set to yes (even if commented), you’re fine. But if it’s been changed, make sure it looks like this:

IgnoreRhosts yes

This ensures SSH will completely ignore .rhosts files, even if they exist – closing off an old and unnecessary security risk.

Disable X11 Forwarding

By default, SSH allows X11 forwarding , which lets you run graphical applications from a remote server and display them locally.

While that can be useful in specific cases (like running a GUI app on a remote server), it’s rarely needed on production servers – and the X11 protocol isn’t built with security in mind.

If you're not using it (and chances are, you're not), it’s best to disable it:

X11Forwarding no

I disable this on every server I deploy. It’s a simple step that helps reduce your server’s attack surface by turning off a feature you likely don’t need.

Disable Agent and TCP Forwarding

SSH supports agent forwarding and TCP forwarding , both useful in certain workflows – but they also come with security risks if left enabled unnecessarily.

Agent Forwarding: It allows you to use your local SSH keys when hopping from one server to another – without copying keys to the first server. It’s convenient, but it can expose your keys if the intermediate server is compromised.
TCP Forwarding (Port Forwarding): It lets you route network traffic between your local machine and remote server – great for tunneling traffic or exposing local services temporarily. But again, if you’re not using it, it’s just one more thing attackers can potentially abuse.

If you don’t explicitly need these features, it’s best to disable them:

AllowAgentForwarding no AllowTcpForwarding no

These are typically enabled by default, so I always turn them off – unless there's a specific reason to keep them on.

Disable Root SSH Access

When you first set up a Linux server, it's common to log in as the root user – but it’s not something you should keep doing.

The root account has full control over the server, so it's easy to make a mistake that causes serious damage. It’s also a common target for brute-force attacks since the username is the same on every server.

Instead, I always recommend logging in with a non-root user that has sudo privileges. This way:

You’re prompted for your password before running administrative commands
Your actions are logged and traceable
The username isn’t obvious to attackers

Here’s how to add one:

adduser username usermod -aG sudo username

Now, locate the PermitRootLogin variable, which is usually commented out by default, and change its value to:

PermitRootLogin no

✅

Now, the root user won’t be able to log in over SSH.

You’ll need to connect using your non-root user – which is much harder for an attacker to guess.

Disable Password Authentication

By default, you can log into a server over SSH using just a username and password. But there's a much more secure way: SSH key authentication.

This method uses a key pair – a public key (stored on the server) and a private key (kept on your machine). When you connect, the server checks if you have the matching private key. If you do, you're in.

This means no passwords to guess – only someone with your private key can log in.

To create a secure key pair, run:

ssh-keygen -b 4096

You’ll be prompted to choose a file path for saving the key – press ENTER to accept the default or enter a custom path to avoid overwriting any existing key.

You’ll also be asked to enter a passphrase. While optional, using a passphrase adds another layer of security and is strongly recommended.

Once done, two files will be created in your ~/.ssh directory:

id_ed25519 – your private key
id_ed25519.pub – your public key

To enable key-based access for your non-root user , copy the public key to the server using:

ssh-copy-id -i ~/.ssh/id_ed25519.pub your.non.root.user@your.server.ip

Once you’ve confirmed that key-based login is working, open your sshd_config file and look for:

#PasswordAuthentication yes

Uncomment it and change the value to:

PasswordAuthentication no

This ensures your server only accepts logins from users with a valid SSH key.

⚠️

Make sure to keep your private key secure – it’s your access pass.

Restrict Who Can SSH Into the Server

Not every user on your server needs SSH access. In fact, the fewer people with SSH access, the better – it tightens security and reduces your attack surface.

There are a couple of ways to limit who can log in over SSH.

Option 1: Use `AllowUsers` (my preferred method)

This method is direct and easy to manage. You specify exactly which users are allowed to connect via SSH – everyone else is denied by default.

At the end of your sshd_config file, add:

AllowUsers ivan john elie

Now, only ivan, john, and elie can log in via SSH.

Option 2: Use `AllowGroups`

You can also create a special group (like sshusers), add your allowed users to it, and then add this to your config:

AllowGroups sshusers

This works well if you want to manage access by group instead of listing users individually.

Restrict Access by IP (Optional but Powerful)

If you have a static IP address (one that doesn't change), you can lock down SSH access even further – only allowing connections from your trusted IP.

You can do this with the same AllowUsers directive:

AllowUsers ivan@203.0.113.45

Now only ivan connecting from IP 203.0.113.45 can access the server.

Want to allow all users , but only from one IP? Use:

AllowUsers *@203.0.113.45

This is a great defense-in-depth tactic – just make sure you’re not on a dynamic IP (which changes), or you might accidentally lock yourself out.

Reload SSH (Don’t Forget!)

Once you've made any changes, always validate your SSH config before applying it:

sudo sshd -t

If there are no errors, reload SSH to apply your changes:

sudo systemctl reload ssh

You can also check the final, active SSH config with:

sudo sshd -T

This lets you confirm that your new settings – like AllowUsers – are active and working.

Bonus: Use `sshd_config.d/` for Cleaner & More Scalable SSH Config

If you're managing multiple servers – or just want a cleaner setup – consider using the sshd_config.d/ directory instead of editing the main config file directly.

Modern Linux distros support config drop-ins via:

/etc/ssh/sshd_config.d/

Instead of changing /etc/ssh/sshd_config, create a new file like this:

sudo vim /etc/ssh/sshd_config.d/hardening.conf

Add your custom settings there – for example:

Port 592 PermitRootLogin no PasswordAuthentication no AllowUsers ivan ClientAliveInterval 60 ClientAliveCountMax 3

This method keeps your config organized, survives server updates, and makes it easier to sync settings across multiple servers with version control or automation.

Conclusion and Final Thoughts

I hope this guide helped you go beyond the basics and gave you a clear, practical path to securing SSH on your Linux server.

Even small tweaks – like changing the SSH port or turning off unused features – can make a big difference in reducing risk and tightening your server’s defenses.

👉

You can check out my full collection of detailed Linux server security guides here – all based on real experience managing my own servers.

💬 Found this guide helpful?

I'd love to hear about your experiences, questions, or ideas in the discussion section below. Your feedback not only helps improve future guides – it helps fellow admins on their own journey.

Prefer a more direct conversation? Feel free to contact me anytime.

How to Install and Use WordPress Locally

Mon, 25 Dec 2023 00:00:00 GMT

Are you planning to create a website in WordPress, but don't want to do it directly on the server?

Discover the smart solution: installing and using WordPress locally.

This approach streamlines your website creation process, allowing you to experiment and refine your site in a safe, offline environment.

In this guide, I will show you step-by-step how to install and use WordPress locally.

Plus, I'll also guide you through the process of taking your site live, ensuring a smooth transition from local development to online presence.

Why WordPress Locally?

Installing WordPress locally means your site lives on your computer.

It's your private playground – unseen by the outside world and search engines until you're ready.

This setup is ideal for creating a new site from scratch. You get to decide when it goes live.

A local WordPress site is a safe zone for experiments. Want to try new themes or plugins? Go ahead, without the fear of breaking your live site.

Updates, new features, even major redesigns – you can test them all in this secure environment.

If you're new to WordPress or coding, think of a local site as a practice area.

You can play around with WordPress files safely, without worrying about losing data or crashing your site.

It's a secure way to learn and try new things.

One of the cool things about a local WordPress site is it doesn't need the internet.

Work on your site anytime, anywhere, regardless of internet connectivity or speed. Perfect for those moments when you're offline.

And here's the best part: it's free.

Beginners can learn all about managing websites and WordPress without paying for hosting or domains. It's practical learning without spending money.

In short, a local WordPress installation is a powerful, risk-free, and cost-effective way to build and refine your WordPress skills and site.

It's all about creating, experimenting, and learning at your own pace, in your own space.

Installing WordPress Locally

To successfully install and use WordPress locally, you need to set up a few key components: a web server, PHP, and a MySQL database.

While it's possible to install these elements manually, it's not the most efficient route.

Manual installation requires individual setup and configuration for each component to create a functional local environment for WordPress.

Fortunately, there are numerous software options available that simplify this process.

These tools allow you to establish a local environment with just a few clicks, often including additional features and add-ons for an enhanced experience.

This approach saves time and reduces complexity, making it an ideal choice for both beginners and experienced users.

In this guide, we're going to use a tool called Local.

I've been using Local for a long time and I absolutely love it.

It streamlines the process, making it accessible and straightforward, even for those new to WordPress.

Go to localwp.com and click on the DOWNLOAD FOR FREE button.

You'll be prompted to select your platform.

If you're a Mac user like me, you have the option to install Local using Homebrew, a convenient alternative.

Next, you'll need to provide some basic information, including your first and last name, work email, and the type of organization you're associated with.

After filling out these details, click on the GET IT NOW! button, and the download will start automatically.

Once the download is complete, open the installation file to install Local on your computer.

Quick Overview

After installing Local, open the application to begin exploring its features.

Initially, you'll be greeted with a page indicating that you haven't created any sites yet – that's perfectly normal, as we'll be creating a new site shortly.

Local's interface is user-friendly, featuring a menu on the left with five distinct sections.

The first section displays the local WordPress sites you create, which will be empty for now.

The second section is for linking Local to WP Engine or Flywheel, enabling you to push your local site live if you have hosting with either of these providers.

The third section introduces Blueprints, a convenient feature for quickly starting new sites using pre-saved settings, including themes and plugins – ideal for replicating a site.

The fourth section contains various add-ons to enhance your Local experience.

Finally, the last section is a valuable resource for support, offering helpful articles and troubleshooting guides.

Preferences

Start by clicking the account icon and selecting Preferences.

This brings up an interface with four sections for customization.

In the Appearance & Behavior section, you can change Local's look and toggle MagicSync on or off, and set your default terminal and browser.

The New site defaults section lets you adjust default settings for new sites, like environment type, WordPress admin email, domain suffix, and file storage location.

Exporting offers an option to exclude certain files during export.

Lastly, the Advanced section is where you can adjust the router mode (which I suggest leaving as is). Here, you can also turn on the Show Develop menu option for debugging tools, and enable Usage and Error Reporting to help improve Local.

I usually keep all the default settings as they are, except for the admin email of WordPress.

Creating a Site

Now, it's time to create your new site.

Begin by clicking on the + Create a new site button.

You'll see a page where you can choose to start from scratch or use a blueprint. Since you don’t have any blueprints yet, select Create a new site.

Next, name your site. If you want, you can click on the Advanced options toggle to customize the domain and path.

For your site's environment, you can pick the Preferred option or set up a custom one. I usually choose the Preferred option.

Then, enter your WordPress username, password, and admin email. Under Advanced options , there’s also the choice to enable WordPress Multisite.

Once you've filled in all the details, click on the Add Site button, and your new site will be set up and ready to go.

Site's Dashboard

Wow, we now have a local WordPress site to play with!

Wasn't that easy? In just a few minutes and clicks, we've set up our very own local environment for WordPress.

You can launch the site in your browser by clicking on the Open Site button.

To directly access the dashboard, click on the WP Admin button. For this, make sure to enable the One-click admin option and select the user you wish to log in with.

Remember, even if you chose an environment while creating the site, you can still change the web server and PHP version later if needed.

You can enable Xdebug if you want. It helps in troubleshooting complex PHP issues.

See the notes sidebar? That's there because I installed an add-on called Notes. Feel free to explore the add-ons section and install any that you find useful.

If you click on the three dots next to the site's name, you will see a list of options.

You can open the site or go directly to the admin dashboard. You have options to stop, restart, or start the site. Additionally, you can clone or export the site.

Creating a blueprint of the site's current state is possible too, allowing you to use it as a template for new sites.

There's also the option to create a new group and move the site into it.

Moreover, you can change the domain, rename the site, or delete it.

Database

The Database section provides information about your database and includes a tool called Adminer.

This tool allows you to connect to and manage your database directly in the browser.

By clicking on the Open Adminer link, a new tab will open in your browser where you can manage the database effectively.

Tools

In the Tools section, you'll find useful tools like Mailhog. It helps you see emails from your WordPress sites.

Just click on the Open Mailhog link, and you can check these emails in your browser.

Another handy tool is Live Links. This allows you to create a shareable link of your local site, which is great for showing your work to clients. You can share the live link online, giving others access to view your site.

However, to use this feature, you need to have a Local account. Feel free to create one. It's free.

Taking Your Site Live

After you've finished editing and preparing your site, it's time to make it live and publicly accessible. But how do you do that? Let me guide you through it.

If you're using WP Engine or Flywheel, you can utilize the Connect feature. However, I'll show you a method that works with any hosting provider.

Firstly, install and activate the All-in-One WP Migration plugin on both your local site and your live site.

All-in-One WP Migration on WordPress.org →

Next, on your local site, navigate to the plugin settings and select Export from the menu.

Replace the URL of your local site with the URL of your live site, then export it. Choose the EXPORT TO FILE option to proceed.

Once the export process is complete, you can download the file that has been generated.

Now, go to your live site, access the plugin settings, and choose Import from the menu.

Then, upload the file you just downloaded. You’ll receive a warning that this process will overwrite the entire website, including the database, media, plugins, and themes. This is expected and exactly what we need to do.

And that's it!

Conclusion and Final Thoughts

I hope this guide has been a valuable resource for learning how to install and use WordPress locally.

By working in a safe, offline environment, you can build and fine-tune your website without the pressure of being live. Once you're ready, taking your site live will be a smooth process with the steps we've covered.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Connect to Your VPS Server

Sun, 12 Nov 2023 00:00:00 GMT

In an earlier guide, I covered the basics of VPS servers and deploying your first one.

Now, it's time to focus on securely connecting to your VPS server.

This simple guide will show you how to connect to your VPS server using Secure Shell (SSH).

What is Secure Shell?

Secure Shell, or SSH, allows two computers to create a secure, direct connection within a potentially insecure network, like the internet.

This is crucial to prevent others from intercepting the data flow and accessing sensitive information.

Before SSH, there were different ways, but they needed to be safer.

Applications like Telnet, Remote Shell, or rlogin were often used, but they had serious security problems.

SSH encrypts the connection between the two computers, enabling one computer to control another.

For us, it's how we connect to our VPS servers and administrate them from our computer.

💡

The development of SSH has influenced other protocols. For instance, the insecure FTP protocol, used for uploading and downloading files, evolved into the SSH File Transfer Protocol (SFTP).

One of SSH's advantages is its compatibility across major operating systems.

Originating as a Unix application, it's inherently implemented in Linux distributions and macOS.

All Linux-based VPS servers let you connect to them directly using SSH, and that's exactly what we'll do in this guide.

💡

People often use ssh as a verb, saying how to ssh into a VPS server instead of how to connect to a VPS server.

Connecting to a VPS Server

To connect to your VPS server, you'll need the root password and the server's IP address.

Many providers either let you set this password while creating the server on their website or send it to you via email along with the server details, like Hetzner does.

The root password provided by providers like Hetzner is temporary. Once you're in your VPS server, you'll need to change it.

👉

New to Hetzner? Use my link to get free credits!

Once you have the server IP and root password, connecting to the VPS server is simple.

Just open the local terminal and type this:

ssh root@

Enter the password, and that's it – you're in!

💡

On Mac or Linux, you can simply use the local terminal. For Windows, I suggest using Git Bash.

If it's your first time connecting to the server (which I assume it is, otherwise, you wouldn't be here reading this guide haha), your terminal might ask if you want to add this server to your list of known hosts - type yes to add it.

You will now be able to run commands on the server directly from your terminal.

Conclusion and Final Thoughts

Simple, right?

I hope this simple guide has been helpful for you.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Getting Started with Your First VPS Server

Thu, 09 Nov 2023 00:00:00 GMT

If you're curious about what a VPS server is and how to deploy your first one, you're in the right place.

In this guide, I'll explain in simple terms what VPS servers are, cool things you can do with them, personal tips from someone who uses them, and I'll even guide you through deploying your very first VPS server for free.

And guess what? After that, I've got some simple steps for what to do next!

Let's get started!

What is a VPS Server?

A VPS, or Virtual Private Server, is like having your own private workspace on a powerful computer that many people share.

Imagine a supercomputer that's used by lots of people, like a library computer that anyone can use. However, the computer is so advanced that it can be split into many smaller, private workspaces.

These smaller workspaces are the VPS servers. Each one runs on its own, just like having your very own computer.

So, a VPS server is like your private workspace within that library computer. Even though it's not a physical computer, it's your own dedicated area with its resources.

Alright, it's time to dive into the tech stuff.

A VPS server allows many users and administrators to work independently on the same hardware.

A powerful physical server machine can host multiple VPS servers, each running on its own operating system.

The management of the central hardware of the physical server is the responsibility of a hypervisor.

Specific software defines the virtual environments and allocates a portion of the server's resources to each VPS server. This includes processor power (CPU), fixed storage space, and RAM (memory).

In fact, it is a virtualized server that operates like a physical server, even though it isn't actually a physical server. It exists within a physical server but has its dedicated resources.

You may have come across VPS providers like Hetzner, Vultr, or DigitalOcean.

These providers have data centers located in various regions where physical servers are housed.

They employ virtualization software to divide these physical servers into VPS servers, each of which can have its own chosen operating system.

This means that users from all over the world can have VPS servers from the same data center on the same physical server, each with different operating systems or software configurations.

Practical Uses of a VPS Server

Now that you know what a VPS server is, let's explore all the cool stuff you can do with it.

Here are some ideas:

Host a Website: You can set up a web server and host your website. Share your ideas, blog, or create an online portfolio.
Run Your Own Email Server: This one is a bit of a fun project! You can create your own email server and manage your emails.
Experiment with Open Source Projects: Dive into open-source software, contribute, or create your own projects. Learn and explore.
Create Your Own Cloud: Store your files, documents, and photos securely in your private cloud storage.
Host Bots: Run your Discord or Telegram bots. Chatbots, game bots, you name it.
Learn and Test: It's also a fantastic playground for learning and testing. Try new software, configurations, and ideas without any worries.

Literally, the possibilities are endless. You can explore, experiment, and create based on your interests and needs.

How does VPS Hosting work?

When it comes to hosting for your website, you have a few choices.

Many people start with shared hosting because it's budget-friendly.

But there's a catch: With shared hosting, your website shares a server with lots of other websites.

You all use the same resources, like the space in a crowded room. This can lead to slower performance and limited control.

Now, there's VPS hosting.

This is like having a room all to yourself. The hosting provider dedicates a VPS server just for your website.

You don't have to share resources with others, so your website performs better. It costs a bit more, but you get a powerful and flexible hosting environment.

Plus, you often get root access, which means you have more control over the server.

💡

Having root access means you have complete control over your server. The root user has the highest level of control on a server.

If you want the best of the best, there's dedicated server hosting.

It's like having a whole building just for you. It's super powerful and can handle massive websites with lots of users.

But it's also super expensive and typically only needed for very large websites.

For most websites, VPS hosting is the sweet spot. You can start small and scale up as needed.

It offers better performance than shared hosting, lower costs than dedicated hosting, and more flexibility than either option.

It's like having the perfect balance for your website's needs.

It's important to note that there's a difference between VPS hosting and having a VPS server.

VPS hosting is the service provided by a hosting company, where they manage the server for you.

In contrast, having a VPS server means you have complete control over the virtual server and are responsible for its management.

I just wanted to clarify all this, what VPS hosting means and its differences from shared and dedicated hosting, as well as the difference between VPS hosting and VPS server, so you don't get confused online.

Here, I'm specifically talking about VPS servers.

How I Use VPS Servers

I mainly use VPS servers for two things: testing and learning. They're like my digital playground.

I also host all my projects on them.

What's cool is that most server providers let you pay by the hour. So, I can deploy a server whenever I want, and after a few hours, I can just delete it.

It's perfect for quick tests. VPS servers are super flexible.

I also use VPS servers to try out new open-source projects. I'm a big fan of open source stuff, and I love experimenting with new things.

VPS servers played a big role in helping me earn my LPIC-1 certificate and pass the exam. I used them to learn and practice, and it was an invaluable resource on my journey to certification.

Deploying Your First VPS Server

When it comes to getting your first VPS server, there are many providers to choose from.

I've always used three VPS providers: Hetzner, Vultr, and DigitalOcean. They all offer similar services and have easy interfaces.

I used them to experiment with new setups, play with open-source projects, and learn more about Linux.

Their servers are powerful and not too expensive, which makes them a great option for beginners who are just starting out.

However, I discovered that Hetzner suits my needs the best and outperforms other providers, so that's the one I use now.

But feel free to pick the provider that works for you.

We'll deploy our first VPS server from Hetzner, but the process is pretty similar with other providers.

👉

New to Hetzner? Use my link to get free credits!

Once you're done signing up, head to the cloud dashboard, and make a new project.

You can name it whatever you like.

Once your project is created, step inside and click Create Resource and chooseServers to deploy your very first VPS server. A new page will open up.

First, pick the server's location. Choose the one that's closest to you.

💡

When you're planning to host a website, consider your visitors. Pick the location nearest to them for a faster website experience.

Next, select the image (operating system) for your VPS server.

I always go with Ubuntu LTS (Long Term Support) version, which is 24.04 as of now. It's reliable and works well for all my projects. But you can choose any image you like.

Then, decide on the type of VPS server you want to deploy. Hetzner offers two types: Shared vCPU and Dedicated vCPU.

Most providers have similar options. To explain, think of it this way:

With dedicated vCPU, you get your own dedicated portion of the physical server's CPU cores. It's like having your own slice of the CPU pie, and what you do won't affect others on the same server.
With shared vCPU, it's like everyone gets a smaller slice of the CPU pie, and the physical cores are shared among multiple VPS servers. Sometimes it's fast, other times it can feel slower if others are using lots of CPU.

For testing and learning, go with shared vCPU.

But if you're running something important, like a live website, dedicated vCPU is the way to go for better performance.

Next, you have three networking options: Public IPv4 , Public IPv6 , and Private networks.

I usually disable IPv6 and stick with IPv4. Some platforms and services don't fully support IPv6 yet, so it's safer to rely on IPv4.

So far, we've set up the essentials for deploying a new VPS server.

There are more options on the page, like adding SSH keys, creating volumes, configuring a firewall, enabling backups, or adding a custom cloud configuration.

We'll skip these for now, as it might be a bit much if you're just starting out.

Your main focus should be on deploying the server.

So, scroll down to the bottom of the page, give your server a name, and then hit Create & Buy now to get your server up and running.

You'll receive an email with your server details. This includes the IP of the server and the root password.

In your project, you'll see a running VPS server with its name, IP, location, and creation time, just like in the picture below.

Just click on the name of your VPS server, and a new page will open up.

There, you can keep an eye on some metrics, change a few options, and check out some VPS settings. Feel free to explore!

By following these steps, you’ve successfully deployed your first VPS server.

What To Do Next

Before diving into configuring your server, installing software, or starting your project, it's crucial to know how to set up and secure your server.

These are essential steps.

Always deploy the server, set it up, and tighten its security – then you're good to go!

I've written detailed guides on these topics, so be sure to check them out!

Conclusion and Final Thoughts

Awesome job reaching the end!

Now you know the basics of VPS servers.

Whether you're planning to host a website, run experiments, or learn something new, you're on the right track.

I hope this guide has been super helpful for you.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Secure WordPress with 2FA

Sat, 04 Nov 2023 00:00:00 GMT

Two-factor authentication (2FA) is one of the most secure methods today to protect your WordPress website from brute-force attacks.

When you use 2FA, you always have to confirm your WordPress backend login with a second method, which is a code that you receive through SMS, email, or an authenticator app like Google Authenticator.

Basically, you need more than just your password to log in.

In my opinion, it is a must-have security procedure that every WordPress website should enable.

In this tutorial, I'll show you how to make your WordPress site safer with 2FA using a free plugin.

Installing Two-Factor Plugin

I use this plugin for all my WordPress sites.

It's lightweight, with over 70,000 people using it, and it has a perfect 5-star rating. I've never had any problems with it.

Two-Factor on WordPress.org →

The first step is super easy.

Just install and activate the plugin, and you're good to go.

Open your WordPress dashboard, head to the plugins page, search for Two-Factor , and then simply install and activate the plugin.

Configuring 2FA

The plugin puts its settings on the user edit page.

When you edit any user on your site, just scroll down until you see the Two-Factor Options section.

You'll find four 2FA methods to choose from: Email, Time-Based One-Time Password (TOTP), FIDO U2F Security Keys, and Backup Verification Codes (Single Use).

You can enable one or more of these 2FA methods as per your preference.

You can choose one of them as your primary method, which you'll use by default when signing in.

If you ever lose access to your primary method, you can change the method during the sign-in process. Think of the other methods as backups to your primary one.

I always enable Email, TOTP, and Backup Verification Codes, with TOTP as my primary method.

If I ever lose access to my phone and can't use the authenticator app, I can simply opt to receive a code via email or use a backup code for easy access.

❗

Your WordPress website needs to be able to send emails via SMTP for you to receive the code by email.

Select your preferred primary method. I suggest TOTP, but the choice is yours.

Make sure to enable at least one backup method. It won't cause any issues if you enable two backup methods, as I do.

So, go ahead and set them up.

Conclusion and Final Thoughts

Adding Two-Factor Authentication to your WordPress site is like putting an extra lock on your door.

It makes sure only you can get in, even if someone has your password.

I hope this tutorial has been of great help to you.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Configure WordPress to Send Emails via SMTP

Fri, 03 Nov 2023 00:00:00 GMT

While WordPress can send emails by default, these emails often fail to reach their intended recipients or end up in the spam folder.

When it comes to sending emails to customers and users through WooCommerce, contact forms, member areas, forums, and more, reliable email delivery is crucial.

In this tutorial, I'll show you how to configure WordPress to use an external SMTP server for sending emails, and the best part is, we'll be using a free service.

Why Use External SMTP Server

If I have an online shop on WordPress, I need a reliable way to send important emails to my customers. These include purchase confirmations, password reset links, and welcome emails when they sign up.

These types of emails are called Transactional Emails , and they're crucial because you always want them to reach your customers on time.

Imagine not getting a password reset link promptly – that wouldn't be ideal, right?

I'm going to explain why using an external SMTP server for sending emails on WordPress is important.

On Shared Hosting

When your WordPress website is on shared hosting, it shares resources with other websites.

Shared hosting means your website is on a server where others have their websites too.

Most shared hosting providers set up their servers to send emails. This means your WordPress website can send emails.

However, because other people share the same server, if someone misuses email sending, it affects all the WordPress websites on that server, including yours.

This might lead to the server's IP address getting blacklisted or blocked, which will affect your email delivery.

On VPS or Dedicated Server

Most server providers block port 25 to prevent misuse for sending spam emails. This means you can't easily configure your server to send emails.

Some providers might open this port if you talk to them about your project, but it's not the best idea. If, for any reason, your server's IP gets blacklisted, it'll affect your email delivery.

Using an external SMTP server from an email service provider (ESP) helps you avoid spam blacklists.

When you send emails through the ESP's server, the recipient's email server checks its IP address, not yours.

So, if your server's IP gets blacklisted, it won't affect your email delivery.

ESPs have a good IP reputation, making your emails less likely to get stuck in blacklists and more likely to reach the recipient.

The "from" Email Address

Normally, WordPress sends notification emails from an address like wordpress@yourdomain.com.

The problem is, that's not a real email address. The trouble is, lots of spam filters think these emails are junk, and they don't even make it to the spam folder. Filters just delete them.

It's really important to use an active email address for sending emails.

This makes your emails look trustworthy like using hello@yourdomain.com or contact@yourdomain.com.

Configuring WordPress to use an external SMTP server for sending emails allows you to change the "from" email address, and I'll cover that in this tutorial.

Security Benefits

By default, WordPress emails lack security. PHP Mail doesn't protect your messages.

This can cause your emails to be treated as spam and not reach your recipient's inbox.

On the other hand, ESPs put in the effort to ensure your emails are secure and reach their destination.

Most ESPs have strong security measures in place, such as SSL or TLS encryption, to keep your emails safe. They invest significantly in maintaining their server reputation, actively watching for any issues.

Free 1000 Monthly Emails with SMTP2GO

There are many ESPs to choose from. Some are paid, and others have both paid and free plans.

This tutorial will use SMTP2GO, which offers a free plan allowing you to send 1000 emails per month.

I’ve been using SMTP2GO for a while now for both production and development projects, and I must say, they’re fantastic.

While I won’t go into a full review, I’d like to highlight some of their excellent features:

No Credit Card Needed: Unlike other ESPs, SMTP2GO doesn’t require your credit card information for signing up for their free plan.
Generous Email Limit: With SMTP2GO’s free plan, you can send up to 1000 emails monthly, which is significantly more than what many other ESPs offer.
User-Friendly Dashboard: Their dashboard is user-friendly and makes it easy to add a domain and set up everything you need.
Exceptional Support: SMTP2GO’s support team is top-notch. They respond quickly and are willing to help with any questions, even if they’re not directly related to their services.

And, it’s worth noting that I’ve never encountered any issues with their service.

SMTP2GO Setup

To get started, you’ll need to sign up for a free account on SMTP2GO.

Visit the smtp2go.com website and click on Try SMTP2GO Free.

A pop-up will appear, prompting you to enter your work email address, not a shared one like @gmail.com or @hotmail.com.

After entering your work email, click on Continue and provide your full name and password.

You’ll receive an activation email to confirm your email address.

Open the email and follow the instructions to activate your account at SMTP2GO, completing the signup process.

After you've activated your account, you'll be taken to your account dashboard, which looks like the picture below.

Click on Add a verified sender , which will open a new page where you can specify the domain from which you want to send emails, as shown in the picture below.

You have two choices: Sender domain and Single sender email.

Opt for Sender domain and click on Add a sender domain.

Enter your desired domain and click Continue with this domain.

❗

As a best practice, your domain should match the one you're using for your WordPress website. Make sure the email address you want to use for sending emails is working. Your domain must have the right MX, DKIM and SPF records set up to send and receive emails properly.

Next, you’ll need to configure DNS records for the domain you’re adding.

SMTP2GO will automatically detect your DNS provider. In my case, it’s Cloudflare.

You’ll need to add just three CNAME records, as illustrated in the picture below.

❗

If you're using Cloudflare, remember not to enable the proxy option.

Once the records are added, click Verify to proceed. Be patient, as it may take a few minutes for the records to take effect.

After that, SMTP2GO will automatically generate an SSL certificate for your domain.

Now that your domain is verified, the final step is to add an SMTP user.

Navigate to the SMTP Users page under the Sending dropdown menu on the left-hand side and select Add SMTP user.

Choose a username and password, and if desired, provide a description, then click on Add SMTP User.

On the same page, you will also find the SMTP server information and the user you’ve just created. Keep this page open because we’ll need it later for the Postfix setup.

With this, we've completed the necessary setup, and we're ready to start configuring WordPress to use the SMTP2GO SMTP server.

Configuring WordPress

Configuring WordPress to use an external SMTP server for sending emails can be done with various plugins.

Some popular options include WP Main SMTP by WPForms, SMTP Mailer, and Easy WP SMTP.

However, in this tutorial, we'll use FluentSMTP, which, in my opinion, is the best SMTP plugin for WordPress.

FluentSMTP is open source, meaning it's free, with no paid features.

It's lightweight and user-friendly, offering in-depth reporting, email logs, the ability to resend emails, and real-time email delivery.

Let's get started.

First, you need to install and activate the FluentSMTP plugin.

FluentSMTP on WordPress.org →

After activation, go to its settings under Settings » FluentSMTP to set it up.

You'll see a welcome message and a list of ESPs to choose from, as shown in the picture below.

Since SMTP2GO isn't listed, select the Other SMTP option.

This will take you to a new page.

Here, you'll need to enter the From Email and From Name.

The From Email you enter will take the place of the default email WordPress uses, which is wordpress@yourdomain.com.

Make sure the email address you use is functional, and that your domain has the correct DNS records in place. Enter a working email address and your name or brand name.

Example: This website's domain is ivansalloum.com. I've added this domain to SMTP2GO and selected the Sender domain option, just like we did in this tutorial. This means I can send emails from any email associated with this domain. In my case, it could be hello@ivansalloum.com, contact@ivansalloum.com, or anyname@ivansalloum.com.

The key is to make sure that any email address I use for sending emails is valid and has an active inbox.

You'll also have three options to enable: Force From Email , Force Sender Name , and Set the return-path to match the From Email.

I, along with FluentSMTP, recommend enabling the first option to ensure all emails are sent from the sender's domain. You can also enable the second option to force the sender's name.

The last option is crucial. Enable it to receive notifications in case an email fails to send due to recipient issues. If this option is disabled, you won't be notified.

Now, it's time to fill in the SMTP Host and SMTP Port fields.

Remember, I mentioned earlier to keep the page open where you added your first SMTP user. You'll find all the necessary information about their SMTP server there.

For SMTP Host , enter:

mail.smtp2go.com

For SMTP Port , enter:

SMTP2GO uses port 2525 for their SMTP server. Other ESPs use different ports, like 587.

If you have trouble with port 2525, SMTP2GO has some other options to use, like 8025, 587, 80, or 25.

Next, choose the TLS encryption option. Don't forget to enable Use Auto TLS and User Authentication.

FluentSMTP, by default, stores the SMTP username and password in the database in an encrypted format, which is highly recommended. This keeps your information secure in the database.

Lastly, enter your SMTP Username and SMTP Password , which you created from the SMTP2GO dashboard, and then click on Save Connection Settings.

Conclusion and Final Thoughts

And there you have it!

Now your WordPress emails are set to hit the inbox reliably with the help of an external SMTP server and the FluentSMTP plugin.

I hope this tutorial has been of great help to you.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

How to Configure Postfix for External SMTP Relay

Sat, 28 Oct 2023 00:00:00 GMT

Postfix serves as a Mail Transfer Agent (MTA).

You can configure it to specialize in sending emails, which is valuable for sending routine email notifications.

In this tutorial, I'll explain how to set up Postfix to send up to 1000 emails per month using a free SMTP relay service.

_I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers _ to get started.

Why Use External SMTP Relay?

Most server providers block port 25 to prevent misuse for sending spam emails.

However, you can work around this limitation by using an external SMTP relay. An external SMTP relay uses a different port, allowing you to send emails externally.

External SMTP relay means your servers don't send emails directly. Instead, it relies on another server, often called a relay host, to send your emails for you.

This approach offers several advantages, including bypassing anti-spam blacklists.

When an email is sent via an external SMTP relay, the recipient's email server checks the relay host's IP address, not yours.

This distinction is crucial because if your server's IP gets blacklisted for any reason, it won't affect your email delivery.

As external SMTP relay services typically maintain a positive IP reputation, your emails are more likely to bypass IP blacklists and successfully reach their intended recipients.

In addition to ensuring reliable email delivery, using SMTP relay services can also save you time and money.

Instead of dealing with the complexities of setting up and maintaining a dedicated SMTP server, opting for a trusted SMTP relay service provides peace of mind.

Consider this scenario: If I have an online shop, I need a reliable way to send important emails to my customers. These include purchase confirmations, password reset links, and welcome emails when they sign up.

These types of emails are called Transactional Emails , and they're crucial because you always want them to reach your customers on time.

Imagine not getting a password reset link promptly – that wouldn't be ideal, right?

Free 1000 Monthly Emails with SMTP2GO

Many Email Service Providers (ESPs) can serve as a relay host for Postfix.

Some of these are paid, while others offer both paid plans and free plans that remain free forever.

This tutorial will use SMTP2GO, which offers a free plan allowing you to send 1000 emails per month.

I've been using SMTP2GO for a while now for both production and development projects, and I must say, they're fantastic.

While I won't go into a full review, I'd like to highlight some of their excellent features:

No Credit Card Needed: Unlike other ESPs, SMTP2GO doesn't require your credit card information for signing up for their free plan.
Generous Email Limit: With SMTP2GO's free plan, you can send up to 1000 emails monthly, which is significantly more than what many other ESPs offer.
User-Friendly Dashboard: Their dashboard is user-friendly and makes it easy to add a domain and set up everything you need.
Exceptional Support: SMTP2GO's support team is top-notch. They respond quickly and are willing to help with any questions, even if they're not directly related to their services.

And, it's worth noting that I've never encountered any issues with their service.

SMTP2GO Setup

To get started, you'll need to sign up for a free account on SMTP2GO.

Visit the smtp2go.com website and click on Try SMTP2GO Free.

A pop-up will appear, prompting you to enter your work email address, not a shared one like @gmail.com or @hotmail.com.

After entering your work email, click on Continue and provide your full name and password.

You'll receive an activation email to confirm your email address.

Open the email and follow the instructions to activate your account at SMTP2GO, completing the signup process.

After you've activated your account, you'll be taken to your account dashboard, which looks like the picture below.

Now, let's prepare everything for our Postfix setup.

Click on Add a verified sender , which will open a new page where you can specify the domain from which you want to send emails, as shown in the picture below.

You have two choices: Sender domain and Single sender email.

Opt for Sender domain and click on Add a sender domain.

❗

Make sure your domain is active, and the email address you want to use for sending emails is working. Your domain must have the right MX, DKIM and SPF records set up to send and receive emails properly.

Next, you'll need to configure DNS records for the domain you're adding.

SMTP2GO will automatically detect your DNS provider. In my case, it's Cloudflare.

You'll need to add just three CNAME records, as illustrated in the picture below.

❗

If you're using Cloudflare, remember not to enable the proxy option.

Once the records are added, click Verify to proceed. Be patient, as it may take a few minutes for the records to take effect.

After that, SMTP2GO will automatically generate an SSL certificate for your domain.

Now that your domain is verified, the final step is to add an SMTP user.

Navigate to the SMTP Users page under the Sending dropdown menu on the left-hand side and select Add SMTP user.

Choose a username and password, and if desired, provide a description, then click on Add SMTP User.

On the same page, you will also find the SMTP server information and the user you've just created. Keep this page open because we'll need it later for the Postfix setup.

With this, we've completed the necessary setup, and we're ready to start configuring Postfix.

Configuring Postfix

Let's start installing and configuring Postfix to use the SMTP2GO SMTP server as a relay host.

First, we'll install Postfix with the following command:

sudo apt install postfix libsasl2-modules

The libsasl2-modules package helps with something called Simple Authentication and Security Layer (SASL), which makes sure programs can securely send emails through a relay server.

It's important when setting up Postfix as a relay host.

During the installation process, a pop-up window will appear, prompting you to select a General mail configuration type , as illustrated in the picture below.

Choose the Internet Site option.

Following this, you'll be asked to input your System mail name , which should be your domain name, as shown in the picture.

❗

Ensure that the domain name matches the one you added to SMTP2GO.

Once you've done this, the installation process will continue until it's complete.

After Postfix is installed, open the main configuration file by using the following command:

sudo vim /etc/postfix/main.cf

At the end of the file, you'll come across a line that starts with relayhost = and has an empty value, which is the default.

In our case, this value should be set to the SMTP2GO SMTP server that we want to use as a relay host.

Remember, I mentioned earlier to keep the page open where you added your first SMTP user. You'll find all the necessary information about their SMTP server there.

Now, set the relayhost value to mail.smtp2go.com:2525, as shown below:

relayhost = mail.smtp2go.com:2525

SMTP2GO uses port 2525 for their SMTP server. Other ESPs use different ports, like 587.

If you have trouble with port 2525, SMTP2GO has some other options to use, like 8025, 587, 80, or 25.

Next, add the following lines to the end of the file:

smtp_sasl_auth_enable = yes smtp_sasl_password_maps = static:yourSMTP2GOUsername:yourSMTP2GOPassword smtp_sasl_security_options = noanonymous smtp_tls_security_level = may header_size_limit = 4096000 relay_destination_concurrency_limit = 20

Replace yourSMTP2GOUsername and yourSMTP2GOPassword with the SMTP user and password you created.

Here's what each directive means:

smtp_sasl_auth_enable: enables SASL authentication.
smtp_sasl_password_maps: tells which username and password to use for SASL authentication.
smtp_sasl_security_options: controls the security options for SASL authentication. noanonymous ensures that authentication isn't anonymous. A valid username and password must be provided to authenticate, just like the SMTP2GO SMTP user you added earlier.
smtp_tls_security_level: sets how safe email encryption should be. "may" means that TLS encryption is optional, and if the receiving server supports it, the connection will be encrypted.
header_size_limit: specifies the maximum allowed size for email headers in bytes.
relay_destination_concurrency_limit: decides how many connections Postfix can make to the relay destination at the same time when sending email.

Don't forget to restart Postfix for the changes to take effect:

sudo systemctl restart postfix

Now, the last thing we need to do is to install the mailutils package, as it provides the mail command used by many programs to send emails.

If you don't install it, several programs won't be able to send you emails, such as Unattended Upgrades.

To install it, use the following command:

sudo apt install mailutils

By following these steps, you've successfully configured Postfix to use SMTP2GO SMTP server as a relay host.

Now, it's time to test our setup.

Testing Our Setup

We can test our Postfix setup by using the mail command to send a test email and check if our server is now able to send emails.

However, it's crucial to note something.

It's very important that your server's hostname matches the domain you have added to SMTP2GO and intend to use.

For example: If my domain is ivansalloum.com, my hostname should be set to server.ivansalloum.com or whatever.ivansalloum.com.

Otherwise, if the domain doesn't match your server's hostname, you will get an error when sending an email which you could examine in the /var/log/mail.log file.

Now, something else you should note.

If we use the mail command directly from the command line, the From header would be user@hostname.

For example: If I use the mail command directly from my terminal using root and my hostname is server.ivansalloum.com, the From header would be root@server.ivansallom.com.

Now the emails should reach their destination, but they might end up in the spam folder.

We only use the mail command for testing.

If the emails end up in the spam folder, it still shows that our server can send emails.

However, emails generated from programs like email alerts and sent from your server will not land in spam.

This is because the From header will show as root@ivansalloum.com, which appears like a real address, even if there's no active mailbox for the root user.

For example, email reports sent from Unattended Upgrades won't land in the spam folder.

Now, let's use the mail command:

sudo mail -s hello@ivansalloum.com

This will send an email from the root user to hello@ivansalloum.com.

If it lands in the spam folder, no problem, as I only want to know if my server is able to send emails.

Conclusion and Final Thoughts

And there you have it!

Now you know how to configure Postfix to be a super reliable email sender using a free SMTP relay service.

I hope this tutorial has been of great help to you.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.

Ivan Salloum

Your First Authoritative DNS Server with CoreDNS

The Big Picture: What We're Building (and What We're Not)

Authoritative vs. Recursive: A Crucial Distinction

Preparing Your Server

Set Your Server's Hostname

Register Your Nameserver's IP (Glue Records)

Free Up Port 53 (Stop systemd-resolved)

Configure Your Firewall (UFW)

Install Docker

Create CoreDNS Directories

Creating Your First Zone File

A Note on Zone File Naming

What Did We Just Create?

Configuring CoreDNS with the Corefile

Docker Compose Configuration

Launching CoreDNS and Local Testing

Delegating Your Subdomain

Public Validation: Confirming Your Delegation

Managing Your DNS Records

Step 1: Edit Your Zone File

Step 2: Increment the Serial Number (Crucial!)

Step 3: Restart the CoreDNS Container

Step 4: Verify Your Changes

Automating Zone Reloads (The Pro Way)

Seeing the Difference: dig in Action

Query 1: A Standard Recursive Query

Query 2: Specifying a Public Resolver

Query 3: Asking Our Authoritative Server for a Public Domain (The "Broken" Test)

Query 4: Asking Our Authoritative Server for a Zone It Controls

Backup and Recovery with Hetzner Snapshots

The Strategy: Protected IPs + Snapshots

My Disaster Recovery Test

Automated Backups & Protection

Conclusion and Final Thoughts

How to Run Beszel for Server Monitoring

Why Beszel? (Context & Goals)

Prep the Server and DNS

Install Beszel Hub + Local Agent with Docker

Create the Compose File

Wire the Local Agent in the Dashboard

Hub-Only Variant

Secure Access with Caddy

Set APP_URL

Add More Servers (Systems)

How Agents Stay Connected

Communication Paths (WebSocket and SSH)

Port 45876 and Firewall Rules

Logs Showing the Fallback in Action

Security Notes (From Beszel’s Docs)

Health Checks

SMTP Configuration

PocketBase Passwords

MFA OTP

Container Metrics

CPU Metrics

Disaster Recovery

My Recovery Checklist

Conclusion and Final Thoughts

How I Built an Adaptive Firewall Setup with UFW and Fail2ban (UFW+)

Author’s Note

Background: The Old Setup and Its Limits

Design Goals and Architecture

1. Smarter Packet Filtering at the Earliest Stage

2. Adaptive Connection and Rate Controls

3. Fail2ban as an Adaptive Enforcement Layer

Architecture Overview

Configuration Walkthrough

Server Preparation

Update and upgrade your server

Create a non-root user

Set your hostname and timezone

Enable UFW and protect SSH access

Allow HTTP and HTTPS traffic

Install a web server for testing (optional)

Install and prepare Fail2ban

Reboot your server

UFW and nftables on Modern Ubuntu

Base Mangle Table Rules (Packet Sanity Filtering)

Edit the UFW before.rules file (IPv4)

Free Up Port 53 (Stop `systemd-resolved`)

Configuring CoreDNS with the `Corefile`

Seeing the Difference: `dig` in Action

Set `APP_URL`

Edit the UFW `before.rules` file (IPv4)

Edit the UFW `before6.rules` file (IPv6)

Drop `INVALID` packets

Drop `NEW` TCP packets without the SYN flag

Edit the UFW `before.rules` file (IPv4)

Edit the UFW `before6.rules` file (IPv6)

Edit the UFW `before.rules` file (IPv4)

Edit the UFW `before6.rules` file (IPv6)

Why I Use `DROP` (silent) instead of `REJECT` / `TCP RST`

Testing `CONNLIMIT` (concurrent connections)