I spent some time debugging a docker host that was running out of space. I came across something that many probably do: docker overlay was taking up a ton of space. Let’s start with a disk free to see the issue we have.
root@vdoppelbock:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 390M 792K 389M 1% /run
/dev/xvda1 117G 6.3G 105G 6% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/259e290a6d59b625ab935b73cafe490b06286ac0b020146b7c6003862ef9fd06/merged
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/b2a401a15e912cff23775f59f0558f454f8003a041843c5845a91cdf09548e94/merged
overlay 117G 6.3G 105G 6% /var/lib/docker/overlay2/5933e3c5e6c9c3bd7e1e688421aaf9c7bf3bf833f7419debe863d71522bcbe58/merged
:/mnt/nfs/container_vols/st_kd8bny 4.5T 54G 4.4T 2% /var/lib/docker/volumes/NFS_syncthing.kd8bny.com/_data
:/mnt/nfs/container_vols/trilium-data 4.4T 326M 4.4T 1% /var/lib/docker/volumes/NFS_trilium-data/_data
tmpfs 390M 0 390M 0% /run/user/0
What the heck is overlay anyway? Well basically it’s docker’s method of storage using content addressable storage. It contains all the data that’s inside the container, to include local volume data.