I spent some time debugging a docker host that was running out of space. I came across something that many probably do: docker overlay was taking up a ton of space. Let’s start with a disk free to see the issue we have.

root@vdoppelbock:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 390M 792K 389M 1% /run
/dev/xvda1 117G 6.3G 105G 6% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/259e290a6d59b625ab935b73cafe490b06286ac0b020146b7c6003862ef9fd06/merged
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/b2a401a15e912cff23775f59f0558f454f8003a041843c5845a91cdf09548e94/merged
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/5933e3c5e6c9c3bd7e1e688421aaf9c7bf3bf833f7419debe863d71522bcbe58/merged
:/mnt/nfs/container_vols/st_kd8bny 4.5T 54G 4.4T 2% /var/lib/docker/volumes/NFS_syncthing.kd8bny.com/_data
:/mnt/nfs/container_vols/trilium-data 4.4T 326M 4.4T 1% /var/lib/docker/volumes/NFS_trilium-data/_data
tmpfs 390M 0 390M 0% /run/user/0

What the heck is overlay anyway? Well basically it’s docker’s method of storage using content addressable storage. It contains all the data that’s inside the container, to include local volume data.

Now what I learned is that docker provides some utilities for the average human.

$docker system df
$docker system df -v

Without the verbose flag we get a nice summary of the space used for the docker system. It even presents space that can be reclaimed with a prune.

root@vdoppelbock:~# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 10 3 1.151GB 936MB (81%)
Containers 3 3 95.06MB 0B (0%)
Local Volumes 3 2 925.8MB 2.27MB (0%)
Build Cache 0 0 0B 0B

Using the verbose flag we get a nice overview of the containers running, local volumes, and build cache.

root@vdoppelbock:~# docker system df -v
Images space usage:

REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS
pihole/pihole latest acf4228d4c8a 4 weeks ago 96MB 8.309MB 87.73MB 1
syncthing/syncthing latest 2071c59bec50 6 weeks ago 41.1MB 8.309MB 32.79MB 1
triliumnext/notes latest 5001a7dab484 2 months ago 254MB 159.5MB 94.07MB 1 < none > < none > d92041aedcbc 2 months ago 92.7MB 8.309MB 84.42MB 0 < none > < none > eab388661907 2 months ago 41.1MB 8.309MB 32.78MB 0 < none > < none > 68b5e4864776 2 months ago 254MB 159.5MB 94.92MB 0 < none > < none > 2859c59a4300 2 months ago 90.2MB 7.401MB 82.77MB 0 < none > < none > 10e1619270b3 3 months ago 359MB 7.834MB 351.1MB 0 < none > < none > 90e4c78a38cf 4 months ago 40.4MB 7.834MB 32.52MB 0 < none > < none > feb8c881adba 4 months ago 90.1MB 7.401MB 82.66MB 0

Containers space usage:

CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES
bf3ef69c62fe triliumnext/notes:latest "docker-entrypoint.s…" 1 0B 12 days ago Up 3 days (healthy) trilium
902edcbe3894 syncthing/syncthing "/bin/entrypoint.sh …" 1 0B 12 days ago Up 3 days (healthy) syncthing.kd8bny.com
57b487f8c0ee pihole/pihole:latest "start.sh" 2 95.1MB 12 days ago Up 3 days (healthy) pihole

Local Volumes space usage:

VOLUME NAME LINKS SIZE
pihole_etc 1 923.5MB
portainer_data 0 2.27MB
pihole_dnsmasq 1 143B

Build cache usage: 0B

CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED

Hope you found this as useful as I did.

–daryl