Whether you're debugging an issue or optimizing server resources, real-time data on CPU usage, memory usage, disk activity, and network traffic is crucial.

This is where dstat shines – a powerful tool that provides detailed, real-time server performance insights.

In this guide, we’ll explore dstat and how you can use it to monitor and analyze your Linux server.

I assume you're working on a properly set-up Ubuntu server. If not, check out my guide on preparing Ubuntu servers to get started.

What is dstat?

dstat is a real-time performance monitoring tool that provides a comprehensive view of server statistics, including CPU usage, memory usage, disk I/O, and network traffic.

Unlike tools like vmstat, iostat, and netstat, which focus on specific resources, dstat combines them all into a single command, making performance analysis easier.

Installation

dstat is not installed by default. To install it, use the following command:

sudo apt install dstat

Once installed, you can verify the installation by running:

dstat --version

Now that dstat is installed, let’s explore how to use it for monitoring your Linux server.

Basic Usage

Running dstat without any options defaults to the -a (all) option, which is equivalent to:

dstat -cdngy

This command displays real-time statistics for:

  • CPU usage (-c)
  • Disk activity (-d)
  • Network traffic (-n)
  • Paging activity (-g)
  • System statistics (-y)

To start monitoring, simply run:

dstat

This will update the server’s performance metrics every second in real time. You can stop it anytime by pressing Ctrl + C.

Here's an example of the output:

----total-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0|   0     0 |7160B 4953B|   0     0 | 167   145 
  0   0 100   0   0|   0     0 |1038B  812B|   0     0 | 108   102 
  0   0  99   0   0|   0     0 |  66B  310B|   0     0 |  33    52 ^C

The dstat output provides a quick overview of key performance metrics to help monitor server activity in real time.

  • CPU usage (usr sys idl wai stl) displays user and system CPU usage, idle time, I/O wait time, and stolen CPU cycles, because it is being used by another virtual machine (VM).
  • Disk activity (read writ) shows the total disk read and write operations.
  • Network traffic (recv send) indicates the amount of data received and sent over the network.
  • Paging (in out) represents memory pages swapped in and out.
  • System metrics (int csw) track system interrupts and context switches.

Additionally, you can enable time/date output with the -t option:

dstat -t

This will display the time and date alongside the performance metrics, giving you a timestamp for each data point. You can also use --time-adv for millisecond precision in the time output.

Monitoring CPU Usage

To get detailed insights into CPU performance, dstat provides options to track overall and per-core usage in real time.

To monitor CPU statistics, use the -c option:

dstat -c

This will display real-time CPU usage, including user (usr), system (sys), idle (idl), wait (wai), and stolen (stl) CPU percentages.

The usr value represents the CPU usage for running user processes (outside the kernel), while the sys value represents the CPU usage for running kernel processes. The idl value indicates the percentage of CPU that is idle. A high wai value indicates that processes are waiting on disk I/O, and stl is relevant in virtualized environments, showing CPU cycles stolen by the hypervisor.

If you want to monitor all available CPU cores dynamically, you can use:

dstat -c -C 0,1,2,3

This will display usage for all CPU cores individually, providing a detailed breakdown of CPU activity.

The -c option will ensure that only CPU performance metrics are shown, while the -C option allows you to specify individual CPU cores by their numbers. The number 0 refers to the first core, as Linux starts counting cores from zero.

You can also use the -f option for a full display of all available CPU cores, instead of specifying each core individually with the -C option.

Additionally, the --cpu-use option focuses only on CPU usage stats and provides a per-core usage breakdown:

dstat --cpu-use

This will show per-core CPU usage in a simplified format, helping you track load distribution across different cores.

To monitor the server load, use the -l option:

dstat -l

This will display a load average over different time intervals (1 min, 5 mins, and 15 mins).

You can also track process-related statistics using the -p option:

dstat -p

This displays the number of runnable, uninterruptible, and newly created processes, helping you analyze CPU load and responsiveness.

  • Runnable are processes ready to execute.
  • Uninterruptible are processes stuck waiting for I/O operations to finish. If too many processes remain in this state, it may indicate disk slowness or other bottlenecks.

For a total count of running processes, use the --proc-count option:

dstat --proc-count

This displays the number of processes, with scal showing the runnable processes and tota showing the total number of processes.

Monitoring Disk Usage

To monitor disk activity, dstat provides options that allow you to track real-time disk read and write operations. You can monitor overall disk usage or focus on specific devices and partitions to analyze disk performance more effectively.

To monitor disk activity in real time, use the -d option:

dstat -d

This will display the total read and write operations on your disk, helping you track how much data is being transferred.

If you have multiple storage devices or partitions, you can specify which one to monitor by using the -D option, followed by the device name, like this:

dstat -d -D sda

This command will display disk activity specifically for the sda device.

The -f option works here as well, providing a full display of all disks, so you don't need to specify them individually.

If you want to go deeper into I/O performance, you can use the -r option to track the number of read and write requests made to the disk. For example, if the server reads 100 files and writes 50 files in one second, this will show the number of I/O operations being initiated:

dstat -r

This will show the number of read and write requests, helping you analyze I/O activity in detail.

Additionally, the --aio option tracks asynchronous I/O operations (non-blocking operations). For example, instead of waiting for one read operation to finish before starting the next, asynchronous I/O allows multiple read operations to happen simultaneously, making the disk more efficient:

dstat --aio

These options help you monitor the frequency and efficiency of disk I/O operations in real time.

Monitoring Memory Usage

To monitor memory usage in real time, dstat provides options to track both physical memory (RAM) and swap space.

To monitor memory usage in real time, use the -m option:

dstat -m

This will display memory usage details, including used, free, buffer, and cache memory.

Buffers temporarily store data being read from or written to disk, improving disk I/O performance by reducing direct disk access. Cache holds frequently accessed files and directory data in RAM, allowing for faster retrieval without needing to read from disk repeatedly. The server dynamically manages this memory, freeing it when needed for active processes.

To monitor swap space usage, use the -s option:

dstat -s

This will display real-time swap usage, showing the amount of swap space that is free and used.

You can also use the -g option to monitor paging activity, which shows how much data is being swapped in and out of memory.

Paging refers to the process of swapping memory pages between physical memory (RAM) and disk storage (swap space) when the server is under memory pressure. This process helps manage memory when the available physical RAM is full.

  • Page in: When data is moved from disk (swap space) back into RAM.
  • Page out: When data is moved from RAM to disk (swap space) to free up memory for other processes.

This is helpful to understand memory management and identify potential performance issues.

Monitoring Network Traffic

To monitor network traffic in real time, dstat provides options to track incoming and outgoing data on your network interfaces.

You can easily track network statistics, such as the amount of data received and sent, and use various options to monitor specific interfaces or the overall network activity.

To monitor network traffic, use the -n option:

dstat -n

This will display real-time statistics for data received and sent over the network interfaces.

If you want to monitor specific network interfaces, such as eth0, or include a summary of all interfaces, you can use the -N option:

dstat -n -N eth0,total

This command will show network statistics for eth0 as well as a total summary for all interfaces combined. This helps you track both individual and overall network performance.

Additionally, to monitor the number of network packets received and transmitted instead of data volume, use the --net-packets option:

dstat --net-packets

This provides a count of packets sent and received, which is useful for analyzing network load and diagnosing packet-related issues.

Conclusion and Final Thoughts

In conclusion, dstat is a versatile tool that helps you monitor real-time server performance by tracking key metrics like CPU, disk, memory, and network activity It’s a great resource for identifying and addressing performance issues.

For more options and advanced usage, check the dstat man page:

man dstat

This will provide additional details to customize the tool for your needs.

If you found value in this guide or have any questions or feedback, please don't hesitate to share your thoughts in the discussion section.

Your input is greatly appreciated, and you can also contact me directly if you prefer.