How to Block INVALID Packets with UFW

Hackers use tools to create TCP packets with unusual, weird flag combinations, known as INVALID packets, capable of causing significant harm.

UFW blocks these INVALID packets by default, but there are still instances where UFW may overlook and fail to block.

In this tutorial, I’ll guide you through the most effective way to block these types of packets using the UFW firewall.

Preparation

To make the most of this guide, ensure you have a properly set up Ubuntu server.

If you don’t have one, consider getting a free VPS server to follow along.

Following along on your own server will enhance your understanding and practical experience.

What are INVALID Packets?

Let me start by explaining what INVALID packets are.

You’re probably familiar with the standard TCP three-way handshake: SYN, SYN-ACK, and ACK.

This process happens every time you visit a website and functions the same for any TCP connection, like SSH.

In a normal connection, it should always begin with the SYN flag after the TCP three-way handshake.

However, abnormal connections don’t start with the SYN flag and don’t follow the TCP three-way handshake.

Malicious actors craft TCP packets with unusual, weird flag combinations, which we aim to block.

How UFW Blocks INVALID Packets

If you look at the /etc/ufw/before.rules file, you’ll notice these two rules:

...
# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
...

These two rules are set to log and block any packets considered INVALID, and they are added by default by UFW.

UFW uses the conntrack module, short for connection tracking, to monitor connections associated with INVALID connection states.

If a connection is flagged as INVALID, it will be logged and blocked.

Let me demonstrate by using Nmap to run an XMAS scan from my MacBook against a server with UFW enabled, which has a rule allowing SSH traffic:

nmap -sX 165.22.65.229

Output
Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 11:14 UTC
Nmap scan report for 165.22.65.229
Host is up (0.0016s latency).
All 1000 scanned ports on 165.22.65.229 are open|filtered
Nmap done: 1 IP address (1 host up) scanned in 27.22 seconds

By default, Nmap scans only the most commonly used 1,000 ports.

The XMAS scan sends INVALID packets containing the FIN, PSH, and URG flags.

As you can see, UFW successfully blocked this scan, resulting in Nmap being unable to determine that port 22 is open.

The output from Nmap that All 1000 scanned ports on 165.22.65.229 are open|filtered suggests that the scan was unsuccessful.

But now, let’s run a Window scan, which involves a significant number of ACK packets:

nmap -sW 165.22.65.229

This should show that port 22 is open or filtered, meaning that UFW was unable to block this scan.

Note: Sometimes, Nmap shows that port 22 is closed. I don’t know the specific reason why, but this still means that the scan was successful.

Blocking INVALID Packets the Right Way

Typically, normal TCP connections should begin with the SYN flag and only the SYN flag.

To further enhance the security of our server, we could add two rules to log and block any new connections that don’t have only the SYN flag set.

Add these two rules below the ones added by default by UFW:

...
-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw-logging-deny
-A ufw-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
...

Don’t forget to add them to the before6.rules file as well:

...
-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ufw6-logging-deny
-A ufw6-before-input -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
...

Reload UFW:

ufw reload

Now, let’s run the Window scan again:

nmap -sW 165.22.65.229

Output
Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-30 13:12 UTC
Nmap scan report for 165.22.65.229
Host is up (0.0017s latency).
All 1000 scanned ports on 165.22.65.229 are filtered
Nmap done: 1 IP address (1 host up) scanned in 21.08 seconds

Great, UFW has effectively blocked this scan, preventing Nmap from determining that port 22 is open.

The PREROUTING Chain

There is one more thing we can do to enhance the way UFW blocks INVALID packets, which is using the PREROUTING chain instead of the INPUT chain.

By leaving everything as is now, INVALID packets will travel through the entire INPUT chain until they reach our rules and get blocked.

However, with the PREROUTING chain, they get blocked before reaching the OUTPUT, FORWARD, or INPUT chains

This way, we are optimizing the performance of UFW.

Since the filter table doesn’t have the PREROUTING chain, we need to use the PREROUTING chain of the mangle table instead.

Add this to the end of the before.rules and before6.rules files after the COMMIT line:

*mangle
:PREROUTING ACCEPT [0:0]
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
COMMIT

Reload UFW:

ufw reload

And that’s it!

Conclusion and Final Thoughts

Great job reaching the end!

I hope this tutorial has been super helpful for you in blocking INVALID packets using the UFW firewall.

However, there’s more to securing Linux servers — advanced measures that I’ve covered comprehensively in my complete server security and hardening guide.

It’s a collection of all security and hardening measures with links to detailed guides for each.

If you found value in this tutorial or have any questions or feedback, please don’t hesitate to share your thoughts in the comments section below.

Your input is greatly appreciated, and you can also contact me directly if you prefer.