Preventing SYN Flood Attacks on Your Linux Server

SYN flood attacks are a common and dangerous type of attack that can overwhelm a server by sending an excessive number of connection requests, ultimately disrupting legitimate traffic and potentially causing the server to go down.

In this guide, I will walk you through various approaches to protect your server from SYN flood attacks.

We’ll explore the use of firewall rules, as well as some kernel parameters that can help mitigate such attacks.

Additionally, we’ll configure Fail2ban to maintain a blacklist of malicious IPs and block them from accessing all ports on the server.

By the end of this guide, you’ll have implemented a robust solution to mitigate SYN flood attacks while ensuring legitimate traffic continues to flow smoothly.

Author's Note

Before we begin, I wanted to share some background.

As someone passionate about Linux server security, I became determined to find a thorough solution for preventing SYN flood attacks.

There are plenty of guides online about preventing SYN flood attacks, but I found most of them incomplete or lacking depth. Many just list a few firewall rules and stop there, while some mention using Fail2ban without offering clear setup instructions.

Frustrated by the lack of comprehensive answers, I decided to explore it myself.

To tackle this, it’s essential to understand both the nature of SYN flood attacks and the basics of TCP, especially the TCP Three-Way Handshake. Once you’re familiar with these, you can start experimenting with ways to block attacks.

Keep in mind that while DoS attacks are manageable, large-scale DDoS attacks are harder to block outright, though you can still make it much more challenging for attackers to succeed. Even basic defenses can keep your server from going down.

Something else worth mentioning is the level of protection provided by cloud service providers, and your own responsibility in securing the services you use.

While providers like Hetzner do offer built-in defenses against large-scale attacks like DDoS, these measures might not always catch smaller SYN flood attempts.

This means that it’s still crucial for you to implement additional protections on your own services, such as your VPS servers, to mitigate these risks.

What is a SYN Flood Attack?

A normal TCP connection begins with a Three-Way Handshake: the client starts by sending a SYN packet, the server responds with a SYN-ACK packet, and then the client completes the process by sending an ACK packet.

Once this handshake is done, the connection is established, and data exchange can begin.

In a SYN flood attack, an attacker takes advantage of this handshake process by sending more SYN packets than the server can handle.

The server responds with SYN-ACK packets but never receives the final ACK, leaving connections half-open.

Since each half-open connection uses server resources, these incomplete connections quickly exhaust the server's capacity to handle new, legitimate traffic. In severe cases, this can cause the server to crash, leaving it unresponsive and inaccessible.

SYN flood attacks can take the form of a DoS attack from a single device or a DDoS attack from multiple devices (often part of a botnet).

This type of attack is also known as a "half-open attack".

How I Tested

To experiment, I created two virtual machines on my MacBook, both running Ubuntu 24.04 server. One acted as the "victim" and the other as the "attacker".

I used hping3 and ApacheBench to simulate SYN flood attacks and excessive traffic.

While ApacheBench is primarily a benchmarking tool for measuring web server performance, it can also be used to overload a server with requests, demonstrating how it might fail under heavy load.

For testing, I installed NGINX on the victim server to keep port 80 open, then I launched a SYN flood attack using hping3 -S -p 80 --flood <ip_address> and monitored the results with tcpdump -nn port 80 on the victim server. 

The --flood option in hping3 sends a continuous stream of SYN packets without completing the TCP handshake, which quickly caused the victim server to become unresponsive.

After a burst of SYN packets, I would eventually see an error that the host was down, and I’d be disconnected from my SSH session. The server simply couldn't keep up.

Additionally, I ran ApacheBench with numerous requests, which raised the victim server's resource usage significantly. No legitimate user would send this many connections in such a short period, so it's a clear indication of an attack.

At this point, I hadn’t applied any firewall rules.

Enabling UFW (Uncomplicated Firewall) on the victim server didn’t significantly help. It lacks built-in defenses against SYN flood attacks.

UFW has a limit rule suitable for SSH, but it’s not practical for production web servers because it restricts IPs that make over 6 connections within 30 seconds, potentially blocking legitimate traffic. For my setup, this was inadequate.

I knew I needed a more refined solution – one that could prevent such attacks without impacting genuine visitors.

After some research, I decided to implement a solution that allows a limited number of SYN packets per IP within a certain timeframe and logs any excess attempts. This way, I could filter out potentially harmful traffic without disrupting normal use.

Instead of simply copying and pasting firewall rules from online sources, I used them as a starting point. From there, I experimented and fine-tuned the rules to develop a solution that was truly effective.

I also wanted a way to maintain a blacklist of IPs, but I ended up using Fail2ban to ban IPs. This worked well with UFW, though it required some custom configuration.

After developing a working solution, I tested it on two real VPS servers from Hetzner, and the results were impressive.

Kernel Settings

Before experimenting with firewall rules, I decided to check if tweaking some kernel parameters could help. It does help, but not significantly.

I’ve already written a guide on kernel hardening, which covers some settings that can mitigate SYN flood attacks. Since these tweaks can be useful, I recommend enabling them, even if they don’t fully stop attacks.

If you open the /etc/sysctl.conf file, you’ll find these three parameters commented out:

#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1
#net.ipv4.tcp_syncookies=1

The net.ipv4.tcp_syncookies parameter is already set to 1 (enabled), which helps mitigate SYN flood attacks by managing half-open connections without consuming too many server resources. While it doesn’t provide much protection, I still recommend keeping it enabled.

You can check if it’s enabled by running:

sudo sysctl net.ipv4.tcp_syncookies

If it’s enabled, leave it as is. If not, uncomment the line in the /etc/sysctl.conf file.

The first two parameters, net.ipv4.conf.default.rp_filter and net.ipv4.conf.all.rp_filter, are set to 2, which means they are in loose mode.

These parameters control reverse path filtering (RP filter), a security feature that helps protect against IP address spoofing.

When enabled, the server checks if it can reach the source address of incoming packets. If it can’t, the packets are dropped.

To enable reverse path filtering in strict mode (which is more secure), change the value to 1 by simply uncommenting them.

Now, there are two additional parameters we can add to optimize how our server handles new connections:

net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 3

The first parameter controls the size of the queue for incoming SYN requests. By default, it’s set to 256 on most systems, which is generally too low. Increasing the backlog allows the server to handle more half-open connections, helping it absorb a higher volume of incoming requests.

For most systems, a range between 4096 and 16384 is recommended unless the system has substantial memory resources.

The second parameter determines how many times the server will resend the SYN-ACK packet if no ACK response is received from the client, as happens in a SYN flood attack.

By default, it’s set to 5, meaning the server waits longer and keeps resources tied up on unresponsive clients. For better SYN flood resistance, reducing this value to 2 or 3 helps free up resources more quickly by limiting the number of retry attempts.

Once you’ve made all changes, save the file, close it, and restart your server for the changes to take effect.

Understanding UFW Rule Files

In the /etc/ufw/ directory, you’ll find three key files: before.rules, before6.rules, and user.rules.

These files control how UFW handles incoming and outgoing traffic, and understanding their roles is crucial.

The before.rules and before6.rules files contain rules that the firewall processes before any user-defined rules are applied.

For instance, if you place a rule to block HTTP traffic in before.rules, it will be processed before any rule in the user.rules file that might allow HTTP traffic.

The rules in the before files take priority, meaning they are executed first, allowing you to enforce more critical security measures, such as rules to block SYN flood attacks.

Specifically, the before.rules file deals with IPv4 traffic, while the before6.rules file handles IPv6 traffic.

The user.rules file contains rules that you add using UFW commands, like sudo ufw allow 80/tcp. When you add a rule via the command line, it gets placed inside the this file.

These rules are processed after those in the before.rules and before6.rules files, making them lower priority.

It's important to note that you should not modify the user.rules file directly, as it could be overwritten by UFW. Instead, any custom rules should be placed in the before.rules or before6.rules files.

Understanding the order in which UFW processes these files is key to managing your firewall properly.

Blocking Invalid Packets

Now it's time to enable the firewall and prepare it for our SYN flood protection rules by blocking invalid packets and ensuring that only normal TCP connections are allowed.

First, ensure that UFW is enabled and that traffic on port 22 is allowed, so you don’t lose access to your server:

sudo ufw limit 22/tcp
sudo ufw enable

Invalid packets include any incoming connections that don’t start with the SYN flag alone.

In a standard TCP Three-Way Handshake, every new connection begins with a SYN packet.

If a TCP connection starts with a different flag or an unusual combination of flags – like those generated by port-scanning tools such as Nmap – these packets should be blocked.

To do this, add the following lines to the end of the before.rules and before6.rules files, right after the last COMMIT line:

*mangle
:PREROUTING ACCEPT [0:0]
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
COMMIT

Now, reload UFW:

sudo ufw reload

These rules added to the mangle table in the PREROUTING chain help to block invalid and potentially malicious packets at the earliest point possible, right when they first arrive at the server.

• The first rule drops any packet that’s considered INVALID by the conntrack (connection tracking) module.
• The second rule blocks TCP packets that are flagged as NEW (indicating new connection attempts) but don’t have the SYN flag set alone.

The mangle table is specifically used for altering packet characteristics or applying special filtering, so it’s ideal for dropping packets that don’t meet the normal connection requirements.

The PREROUTING chain is one of the first chains that incoming packets go through as soon as they reach the server. By applying our rules here, we can block invalid packets immediately, before they reach other parts of the firewall.

This reduces the load on our firewall and helps ensure that only valid traffic is processed further.

SYN Flood Protection Rules

Now it’s time to add the firewall rules that will help protect us from SYN flood attacks.

I’ll assume you already have a firewall enabled with only SSH traffic allowed, along with the rules to block invalid packets.

Since we’ve already used the limit rule for SSH, there's no need to add additional protection against SYN flood attacks for SSH, as it’s already protected by UFW.

But, if you prefer, you can use these same rules to protect SSH as well, without using the limit rule. The rules I’m about to cover will apply to any service that uses TCP.

Approach One

I started with these rules, adding them to the end of the before.rules file, just before the last COMMIT line:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m limit --limit 10/second --limit-burst 20 -j ACCEPT
-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m limit --limit 10/second --limit-burst 20 -j LOG --log-prefix "[UFW SYN Flood Detected] "
-A ufw-before-input -p tcp --syn --dport 80 -j DROP

These rules help protect your server from SYN flood attacks by limiting the rate of incoming SYN packets to port 80.

• Step 1: The first rule allows up to 10 SYN packets per second from a single IP address, with an initial burst of 20 packets.
• Step 2: The second rule logs any SYN packets that exceed this limit, tagging them with the prefix [UFW SYN Flood Detected] so you can monitor potential attack attempts.
• Step 3: The final rule drops any SYN packets that exceed the rate limit, ensuring that the server only processes legitimate connection attempts while blocking excessive traffic.

Together, these rules ensure that your server can handle legitimate traffic while blocking excessive SYN requests that could lead to a SYN flood attack.

I used the ufw-before-input chain directly without adding any new chains.

The ufw-before-input chain is a set of rules that are processed before any user-defined rules. It is used to filter incoming traffic before it reaches the server, allowing you to set up default security measures.

These rules are applied early in the firewall process, ensuring that potentially harmful traffic is stopped before it reaches any services or applications running on the server.

Since our first rule allows traffic on port 80 as long as it doesn't exceed the specified limits, there is no need to allow HTTP traffic from the command line. As mentioned earlier, the before.rules file is executed before any user-defined rules.

We should also add rules to the before6.rules file in the same way, but we need to change the chain from ufw-before-input to ufw6-before-input.

Now, if I use hping3 to simulate an attack, our rules will keep dropping packets. If you check the /var/log/syslog file, you'll see it continuously flooding with logs containing the [UFW SYN Flood Detected] prefix until the attack is stopped.

We did it. We successfully blocked the SYN flood attack. The server usage only increases slightly, which is perfect.

Approach Two

The initial rules worked well and successfully blocked a SYN flood attack, but they had a major drawback.

I used the limit module, which sets a global limit. This means that if one IP address initiates an attack, all other IPs are also blocked.

Essentially, we were limiting the number of SYN packets globally across all IP addresses, which is not ideal. Instead, we need to set limits on a per-IP basis.

Moreover, a global limit of 10 SYN packets per second is not ideal for a web server.

To address this, I replaced the limit module with the hashlimit module, which allows setting limits for each individual IP address. I then updated the rules as follows:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP
-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] "
-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT

Our rules are now applied on a per-IP basis, preventing all users from being penalized due to a single misbehaving IP. This approach offers better scalability for handling legitimate traffic during attacks.

Key parameters explained:

• --hashlimit-mode srcip:
  Applies the rule based on the source IP address, ensuring that each user is treated individually.
• --hashlimit-srcmask 32:
  Limits are applied specifically to each IP address (/32 refers to a single IP). If you wanted to group IPs (e.g., per subnet), you could reduce the mask size (e.g., --hashlimit-srcmask 24).

I initiated another attack using hping3 and, at the same time, used the curl command to check if I could access the victim server's IP.

Sure enough, I got the NGINX default page – it’s working!

Approach Three

Next, I wanted a way to maintain a blacklist of IPs that had previously attacked my server.

To achieve this, I combined the hashlimit module with the recent module to dynamically manage a list of IPs. This allows us to add IPs that exceed our limits to the blacklist and drop packets from IPs already on the blacklist.

Our updated rules now look like this:

-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -m recent --name blacklist --set --rsource -j DROP
-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] "
-A ufw-before-input -m recent --name blacklist --rcheck --seconds 300 --hitcount 1 --rsource -j DROP
-A ufw-before-input -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT

Now, whenever an IP exceeds our limits, it gets added to the blacklist and you can view the list of IPs using:

sudo cat /proc/net/xt_recent/blacklist

Once an IP is blacklisted, any packets from that IP are dropped for five minutes.

If an attacker keeps sending packets frequently, they’ll remain on the blacklist indefinitely because the timer keeps resetting. If a legitimate user is mistakenly blacklisted, they’ll automatically be removed after the timeout period if they stop sending packets.

I initiated another attack using hping3, and once my IP was blacklisted, I used the curl command to check if I could access the victim server's IP.

As expected, I couldn't. I had to wait five minutes for the blacklist timeout before I could access the default NGINX page again. Perfect!

Final Solution

For our final solution, I aim to protect both the HTTP and HTTPS ports from SYN flood attacks and use Fail2ban to block access to all ports on the server and maintain a blacklist.

When using the recent module, the blacklist of IPs doesn’t persist after a server reboot because it is stored only in memory.

For me, this isn’t a major issue since, after a reboot, the server simply starts collecting IPs again that exceed our limits. So, having the blacklist cleared upon reboot isn’t a significant concern.

However, I considered using Fail2ban alongside our rules to maintain a persistent blacklist of IPs because it offers a more robust way to block and unblock IPs, as well as manage the blacklist effectively.

By configuring Fail2ban to monitor the /var/log/syslog file for entries with the log prefix we defined in our rules, we can block any IP that appears, for example, five times, from accessing all ports on the server.

And instead of blacklisting IPs immediately when they trigger our rules, Fail2ban will only block them if they appear five times in the log file.

This ensures that legitimate IPs are not blacklisted if they mistakenly or for some reason exceed the limit, unlike when using the recent module.

Now, install Fail2ban using:

sudo apt install fail2ban

Next, we need to create a new filter.

Go to the /etc/fail2ban/filter.d directory and create a file named synflood.conf with the following contents:

[Definition]
failregex = .*UFW SYN Flood Detected.*SRC=<HOST>.*DPT=\d+.*
ignoreregex =

This will capture log entries for both port 80 and port 443, effectively targeting any IP logged with our specified prefix.

Now, go back to the /etc/fail2ban directory and create a local copy of the jail.conf file:

sudo cp jail.conf jail.local

This is important because you should not modify the jail.conf file directly, as it may be overwritten during an update.

Open the jail.local file and, under the # JAILS section, add the following:

[synflood]
enabled  = true
filter   = synflood
action = iptables[type=allports, name=synflood, chain=fail2ban, protocol=tcp]
logpath  = /var/log/syslog
maxretry = 5
findtime = 600
bantime  = 86400

We are creating a new jail called synflood that will block IPs from accessing all ports on the server if they appear five times in the syslog file within ten minutes, for a duration of one day.

As you may have noticed, I’m using Iptables to block IPs instead of UFW, by adding them to a new chain called fail2ban. The reason for this is that if we use UFW, it will block IPs at the user level by adding them to the user.rules file.

However, this won’t take effect because the before.rules file is processed first.

You can also block traffic only on ports 80 and 443 by using the multiport option, like this:

action = iptables[type=multiport, name=synflood, chain=fail2ban, port="http,https", protocol=tcp]

Now, before restarting Fail2ban, there’s something we need to change.

Open the before.rules file and create two new chains placing them under the # End required lines line:

:SYN_FLOOD_PROTECTION - [0:0]
:fail2ban - [0:0]

Change our rules to this:

-A ufw-before-input -j fail2ban
# (fail2ban will add its IP block rules here)
-A fail2ban -j RETURN
-A ufw-before-input -p tcp --syn --dport 80 -j SYN_FLOOD_PROTECTION
-A ufw-before-input -p tcp --syn --dport 443 -j SYN_FLOOD_PROTECTION
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] "
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j DROP
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -m hashlimit --hashlimit-name http_limit --hashlimit-above 10/second --hashlimit-burst 20 --hashlimit-mode srcip --hashlimit-srcmask 32 -j LOG --log-prefix "[UFW SYN Flood Detected] "
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT
-A SYN_FLOOD_PROTECTION -p tcp --syn --dport 443 -m conntrack --ctstate NEW -j ACCEPT

First, we redirect traffic from the ufw-before-input chain to the fail2ban chain.

The -A fail2ban -j RETURN rule exits the fail2ban chain and moves to the next rule, which redirects traffic to our SYN_FLOOD_PROTECTION chain.

In the fail2ban chain, Iptables adds the rules for blocking IPs that have been blacklisted. You won’t see these rules in the file directly because Iptables can't write to the file, but I’ve added a comment for clarity.

Essentially, UFW first checks the fail2ban chain and denies traffic from any IPs listed there. Once it reaches the RETURN rule, traffic moves to the SYN_FLOOD_PROTECTION chain.

The SYN_FLOOD_PROTECTION chain will then drop traffic from IPs exceeding our set limits and log those IPs.

If an IP doesn’t exceed the limits, it’s allowed access, thanks to the final rules that permit traffic from IPs not caught by any previous rule.

Finally, restart Fail2ban:

sudo systemctl restart fail2ban

To check the status of our new jail, we can use the command:

sudo fail2ban-client status synflood

This command shows details about the synflood jail we created and enabled. It displays the number of currently blocked IPs, the total number of blocked IPs, and a list of the IPs that have been blocked.

Let's inspect the content of the custom chain we created:

sudo iptables -L fail2ban -v -n

You will notice two rules inside this chain.

The first redirects traffic to a new chain called f2b-synflood, which Fail2ban automatically creates to manage and organize firewall rules. This ensures that if the same chain is used for another jail, the rules remain separate.

The second rule allows any traffic not handled by the f2b-synflood chain to proceed.

To view the block rules that Fail2ban has added, examine the contents of the f2b-synflood chain using the command:

sudo iptables -L f2b-synflood -v -n

Now, if I initiate an attack using hping3, Fail2ban will detect the attack and add a rule to block my IP from accessing all ports:

Chain f2b-synflood (1 references)
 pkts bytes target     prot opt in     out     source               destination         
1657K   66M REJECT     0    --  *      *       192.168.64.7         0.0.0.0/0            reject-with icmp-port-unreachable
 445K   18M RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

And indeed, Fail2ban successfully caught me.

If I want to unblock my IP, I can simply use the command:

sudo fail2ban-client set synflood unbanip 192.168.64.7

It's simpler compared to using the recent module.

You can also check the IPs that Fail2ban has identified in the log file but has not yet blocked (because they are still under the limit) by using the sudo grep synflood /var/log/fail2ban.log command.

What I like about this solution is that our Fail2ban setup targets any IP logged with our specified prefix. This means you can add new rules, such as for SSH, without modifying the Fail2ban configuration – simply use the same prefix for all rules.

Key Consideration

I'd like to highlight an important point I mentioned earlier in this guide.

Our current setup works well for blocking small to medium-sized attacks, as we allow a certain number of SYN packets per IP – 10, with an initial burst of 20.

However, what if an attacker launches a DDoS attack using multiple IP addresses, each sending fewer packets than our limits?

I haven’t tested this scenario, so I can’t confirm how it would behave, but it’s a question worth considering.

While each individual IP stays under the limit, the total volume of SYN packets could still be massive, and our firewall rules wouldn’t be triggered.

That's why I recommend using a secure server provider like Hetzner, which has DDoS protection configured at the cloud level.

Additionally, if you're hosting an application with NGINX, consider using its built-in rate limiting features for an added layer of defense.

You can also use a CDN service like Cloudflare, which provides DDoS protection.

Conclusion and Final Thoughts

In this guide, we explored several approaches to protect a server from SYN flood attacks, including using firewall rules, the limit, hashlimit and recent modules, and Fail2ban for dynamic IP blocking.

Although it was a challenging process, we successfully enhanced the server's security, enabling it to handle SYN flood attacks while still allowing legitimate traffic.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.