Hackers use tools to create TCP packets with unusual, weird flag combinations, known as Invalid Packets, capable of causing significant harm.

UFW blocks these invalid packets by default, but there are still instances where it may overlook and fail to block.

In this tutorial, I'll guide you through the most effective way to block these types of packets.

Preparation

To make the most of this guide, ensure you have a properly set up Ubuntu server.

👉
Check out my guide on preparing Ubuntu servers to ensure your server is properly set up.

If you don’t have one, consider getting a free VPS server to follow along.

What are Invalid Packets?

Invalid packets include any incoming connections that don’t start with the SYN flag alone.

In a standard TCP Three-Way Handshake, every new connection begins with a SYN packet.

If a TCP connection starts with a different flag or an unusual combination of flags – like those generated by port-scanning tools such as Nmap – these packets should be blocked.

How UFW Blocks Invalid Packets

If you look at the /etc/ufw/before.rules file, you'll notice these two rules:

# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP

These two rules are set to log and block any packets considered INVALID, and they are added by default by UFW.

UFW uses the conntrack module, short for connection tracking, to monitor connections associated with INVALID connection states.

Let me demonstrate by using Nmap to run an XMAS scan from my MacBook against a server with UFW enabled, which has a rule allowing SSH traffic:

nmap -sX 165.22.65.229

Output:

Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 11:14 UTC
Nmap scan report for 165.22.65.229
Host is up (0.0016s latency).
All 1000 scanned ports on 165.22.65.229 are open|filtered
Nmap done: 1 IP address (1 host up) scanned in 27.22 seconds

By default, Nmap scans only the most commonly used 1,000 ports. The XMAS scan sends invalid packets containing the FIN, PSH, and URG flags.

As you can see, UFW successfully blocked this scan, resulting in Nmap being unable to determine that port 22 is open.

The output from Nmap that All 1000 scanned ports on 165.22.65.229 are open|filtered suggests that the scan was unsuccessful.

But now, let's run a Window scan, which involves a significant number of ACK packets:

nmap -sW 165.22.65.229

This should show that port 22 is open or filtered, meaning that UFW was unable to block this scan.

Blocking Invalid Packets the Right Way

To further enhance the security of our server, we could add two rules to log and block any new connections that don't have only the SYN flag set.

Add these two rules below the ones added by default by UFW:

-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw-logging-deny
-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Don't forget to add them to the before6.rules file as well:

-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw6-logging-deny
-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Now, reload UFW:

sudo ufw reload

These rules help to block invalid and potentially malicious packets:

  • The first rule drops any packet that’s considered INVALID by the conntrack module.
  • The second rule blocks TCP packets that are flagged as NEW (indicating new connection attempts) but don’t have the SYN flag set alone.

Okay, let's run the Window scan again:

nmap -sW 165.22.65.229

Output:

Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 13:12 UTC
Nmap scan report for 165.22.65.229
Host is up (0.0017s latency).
All 1000 scanned ports on 165.22.65.229 are filtered
Nmap done: 1 IP address (1 host up) scanned in 21.08 seconds

Great, UFW has effectively blocked this scan, preventing Nmap from determining that port 22 is open.

The PREROUTING Chain

There is one more thing we can do to enhance the way UFW blocks invalid packets, which is using the PREROUTING chain instead of the INPUT chain.

By leaving everything as is now, invalid packets will travel through the entire INPUT chain until they reach our rules and get blocked.

However, with the PREROUTING chain, they get blocked before reaching the OUTPUT, FORWARD, or INPUT chains. This way, we are optimizing the performance of our firewall.

Since the filter table doesn't have the PREROUTING chain, we need to use the PREROUTING chain of the mangle table instead.

To do this, add the following lines to the end of the before.rules and before6.rules files, right after the last COMMIT line:

*mangle
:PREROUTING ACCEPT [0:0]
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
COMMIT

Reload UFW:

sudo ufw reload

The mangle table is specifically used for altering packet characteristics or applying special filtering, so it’s ideal for dropping packets that don’t meet the normal connection requirements.

The PREROUTING chain is one of the first chains that incoming packets go through as soon as they reach the server. By applying our rules here, we can block invalid packets immediately, before they reach other parts of the firewall.

This reduces the load on our firewall and helps ensure that only valid traffic is processed further.

Conclusion and Final Thoughts

Great job reaching the end!

I hope this tutorial has helped you to block invalid packets and enhance your server's security.

👉
You can find the full collection of detailed Linux server security guides here.

If you found value in this tutorial or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.