HostDiskWillFillIn24Hours #

Meaning #

Filesystem is predicted to run out of space within the next 24 hours at current write rate

Alert Rule

alert: HostDiskWillFillIn24Hours
annotations:
  description: |-
    Filesystem is predicted to run out of space within the next 24 hours at current write rate
      VALUE = {{ $value }}
      LABELS = {{ $labels }}    
  runbook: https://srerun.github.io/prometheus-alerts/runbooks/node-exporter/hostdiskwillfillin24hours/
  summary: Host disk will fill in 24 hours (instance {{ $labels.instance }})
expr: ((node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes &lt; 10 and ON
  (instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~&#34;tmpfs&#34;}[1h],
  24 * 3600) &lt; 0 and ON (instance, device, mountpoint) node_filesystem_readonly ==
  0) * on(instance) group_left (nodename) node_uname_info{nodename=~&#34;.&#43;&#34;}
for: 2m
labels:
  severity: warning

Impact #

The impact of a disk filling up can range from applications failing to the host panicing and shutting down depending on which disk is filling and the operating system.

Diagnosis #

The alert will provide the host and filesystem that is in danger of filling up. For Windows machines, login and check the storage manager for the filesystem indicated.

On Linux machines, login and run

df -h

this will show the utilization of all mounted filesystems.

To determine where the majority of the utilization is, change into the mounted directory and run:

sudo df -h *

This will list out each directory and the amount of space it is using.

Mitigation #

There are two options to mitigate the issue.

Add more space to the filesystem, eg.
- grow the disk on a vm
- add an additional drive and span them on the filesystem
Free up space on the affected drive
- using the output of the df -h command, locate the directories using the most space
- check the files in those directories to determine what can be deleted