Docker error : no space left on device

Yesterday morning I was informed the KNoT cloud instance, which we run in EC2 machines on AWS, were down. We don’t have robust monitoring services yet and I usually enter the cluster manager instance to discover why the services are down and to bring them up again.

The first thing I did was to inspect one of the stopped containers and I saw the error “no space left on device”.

docker service ps <container_id> --no-trunc

ld6rde7xuv2pwsg8270b7qo21    \_ authenticator  <image>   <address>    Shutdown   Rejected <time>   "symlink <path> <overlay>: no space left on device"

After that, maybe the obvious action to take is to analyze the instance’s disk space:

df -h

/dev/xvda1       16G  8.8G  7.3G  55% /

As we can see, the disk space isn’t the problem but the error is clearly indicating it’s a space problem. So, after searching quickly the docker overlay directories became the main target and I noticed the container log files were too big for the expected overlay size. For example, one of these files had 2.2G of size.

sudo du -d1 -h /var/lib/docker/containers | sort -h

2.2G	/var/lib/docker/containers/c07ca3a6f88c0b88718c7...

To solve this problem I followed these steps:

Decide if the log files must be truncated or entirely cleared.

Truncate: a better option when you are concerned about the containers that are running at the moment.

  sudo sh -c "truncate -s 0 /var/lib/docker/containers/*/*-json.log"

Clear: a better option you aren’t concerned about the state of the running containers.

  cat /dev/null > /var/lib/docker/containers/*/*-json.log`

Update the docker configuration file to set limits on log size and enable it to be rotated. Besides, I also set a limit on the number of files stored after rotation.

To do this, edit the file /etc/sysconfig/docker and modify the OPTIONS variable to add these flags: --log-opt max-size=<size> --log-opt max-file=<number>