Get Notified About Linux Server Running Out of Disk Space

I self-host all my websites through cloud server providers like Linode and DigitalOcean.

Keeping the server up and running is my responsibility. While investigating, I found that a few times servers became unresponsive and the services went down because the disk usage was 100%.

In fact, my Discourse server often runs out of disk space due to huge Docker log files and automated backups.

This is the reason I have set up disk usage alerts on most of my production servers. If 80% of disk space is full, I get email notifications. This gives me enough time to act and delete unnecessary files to free up space on the server.

In this article, I'll share a few ways you can also set up disk space alert on your Linux servers:

• Go old school and use a bash script that watches the disk space and sends an email
• Enter the modern world and use a tool like Netdata for effortless monitoring
• Use your cloud server's built-in alert system (if you use any)

Method 1: The classic Linux way

This method is shared by Nixcraft and it works well. Although, it may seem like a rather complicated set up for newbies. Here are the steps involved:

• Install and setup mail utilities on your server
• Write a bash script to check disk space usage (you can copy the code and run it to see if it works)
• Set up a cronjob to run the above bash script periodically

Here's the bash script that will check and alert if the disk space utilization crosses the threshold of 80%. Let's call it disk_alert.sh.

#!/bin/sh
# Purpose: Monitor Linux disk space and send an email alert to $ADMIN
ALERT=80 # alert level 
ADMIN="you@your-email" # dev/sysadmin email ID
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read -r output;
do
  echo "$output"
  usep=$(echo "$output" | awk '{ print $1}' | cut -d'%' -f1 )
  partition=$(echo "$output" | awk '{ print $2 }' )
  if [ $usep -ge $ALERT ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
    mail -s "Alert: Almost out of disk space $usep%" "$ADMIN"
  fi
done

There are ways to send emails from a Linux server. You can use SMTP to send emails from the terminal or use one of the methods discussed here:

How to Send Emails from Ubuntu Command Line

Here’s a detailed tutorial that shows you the steps for using a Gmail like service to send emails from your Ubuntu server.

Learn UbuntuPratham Patel

Now, you can copy this disk_alert.sh file to /etc/cron.daily/ if you want to run it once a day, daily.

You need to give execute permission to this shell script:

chmod +x /etc/cron.daily/disk_alert.sh

Method 2: The modern way with Netdata

Netdata is a complete monitoring solution. It is open source software; you can either self host it or use their hosted service. You get all kinds of performance metrics in a pretty dashboard and this includes disk space utilization.

The good thing is that your data is never stored on Netdata cloud. The dashboard is only a viewer and it fetches the data stored on your system (by Netdata client).

Their community program allows up to 5 connected servers. This suffices my needs for now.

Here's how it works. Sign up for Netdata and log in to your account. In the empty space there, you'll be promoted to 'connect nodes'. When you click that, you can choose the platform where you want to install Netdata and copy the provided code.

For Linux, it's a script that is downloaded using wget or curl and then it installs Netdata agent on your system. It doesn't require sudo access anywhere.

More details can be found here.

Netdata Agent Installation | Learn Netdata

Netdata is very flexible and can be used to monitor all kinds of infrastructure. Read more about possible Deployment guides to understand what better suites your needs.

Netdata Logo

Once your server is connected, it starts fetching various system monitoring data and display it in a dashboard.

You can either scroll down or click on the Mount Points to see the disk space utilization. That Disks options shows the I/O operation metrics of disk.

You can see the disk space usage in a pi-chart graph. If you hover the mouse over it, it shows a few options that includes alert configuration.

There are two threshold values; warning and critical. You can set them to whatever values you deem fit. You can even set at what interval you want it to check disk usage.

There is nothing more to be done. You'll get timely notification in the email address associated with the Netdata account.

Here's an example email for a different alert.

Conclusion

Some cloud server providers like DigitalOcean also allow you to easily set up system resource alerts.

You may want to check the documentation of your cloud service provider to see whether there is such an option available.

I rely on Netdata because it is versatile and works out of the box for monitoring a huge range of performance metrics.

Now, I let you decide which of the above methods you would prefer. If you have some other suggestion, please let me know in the comment section.