I used Docker for AWS to create a Docker swarm cluster. In every node of this cluster, I found Docker root dir is /var/lib/docker, however, I cannot find this directory in file system.
docker info shows this:
~ $ docker info
Containers: 7
Running: 7
Paused: 0
Stopped: 0
Images: 9
Server Version: 17.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: awslogs
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: k8nw4j8bt0mhip541vurvd785
Is Manager: false
Node Address: 10.x.y.z
Manager Addresses:
10.x.y.z:2377
10.x.y.z:2377
10.x.y.z:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.36-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.293GiB
Name: ip-10-x-y-z.us-west-2.compute.internal
ID: 4YZB:UYHX:REW2:ENBZ:AAEW:BS6T:HB5B:OQCE:FRBU:DPXX:SJWV:GRGL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 163
Goroutines: 233
System Time: 2018-06-13T07:04:01.824290758Z
EventsListeners: 5
Registry: https://index.docker.io/v1/
Labels:
os=linux
region=us-west-2
availability_zone=us-west-2a
instance_type=m3.large
node_type=worker
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
It clearly shows the Docker Root Dir is /var/lib/docker, however, when I use ls, it says the directory cannot be found:
~ $ sudo ls /var/lib/docker
ls: /var/lib/docker: No such file or directory
~ $ sudo ls -al /var/lib/
total 20
drwxr-xr-x 5 root root 4096 Jun 25 2017 .
drwxr-xr-x 1 root root 4096 Jun 27 2017 ..
drwxr-xr-x 2 root root 4096 Jun 25 2017 apk
drwxr-xr-x 2 root root 4096 Jun 25 2017 misc
drwxr-xr-x 2 root root 4096 Jun 25 2017 udhcpd
About Docker's disk usage, there are over 2.3GB disk used by Docker
/ $ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 9 6 2.361GB 1.702GB (72%)
Containers 7 7 665.1kB 0B (0%)
Local Volumes 6 6 19.85MB 0B (0%)
About file system usage, df command shows 2.9GB space is used, which is expected since Docker alone consumes over 2.3GB.
/ $ sudo df -h
Filesystem Size Used Available Use% Mounted on
overlay 19.7G 2.9G 15.8G 15% /
tmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.6G 0 3.6G 0% /sys/fs/cgroup
tmpfs 3.6G 161.2M 3.5G 4% /etc/group
tmpfs 3.6G 161.2M 3.5G 4% /etc/passwd
tmpfs 3.6G 161.2M 3.5G 4% /etc/shadow
/dev/xvdb1 19.7G 2.9G 15.8G 15% /etc/ssh
tmpfs 3.6G 161.2M 3.5G 4% /home/docker
/dev/xvdb1 19.7G 2.9G 15.8G 15% /var/log
/dev/xvdb1 19.7G 2.9G 15.8G 15% /etc/resolv.conf
/dev/xvdb1 19.7G 2.9G 15.8G 15% /etc/hostname
/dev/xvdb1 19.7G 2.9G 15.8G 15% /etc/hosts
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 746.8M 916.0K 745.9M 0% /var/run/docker.sock
tmpfs 3.6G 161.2M 3.5G 4% /usr/bin/docker
tmpfs 3.6G 0 3.6G 0% /proc/kcore
tmpfs 3.6G 0 3.6G 0% /proc/timer_list
tmpfs 3.6G 0 3.6G 0% /proc/sched_debug
tmpfs 3.6G 0 3.6G 0% /sys/firmware
At the same time, about disk usage:
/ $ sudo du -h -d 1
0 ./sys
31.5M ./usr
1.8M ./etc
0 ./proc
12.0K ./home
216.0K ./sbin
4.0K ./tmp
8.0K ./run
4.0K ./root
1.4M ./bin
114.3M ./var
4.0K ./mnt
16.0K ./media
2.8M ./lib
0 ./dev
4.0K ./srv
152.2M .
du reports only 152.2M disk is used, and /var folder has only 114.3M used, and it seems du fails to find the /var/lib/docker as well, and Docker stores its data (2.3+GB) in some way du cannot find.
If I used docker inspect to inspect one of the running container on this node, it shows multiple folders are under /var/lib/docker.
Can anyone shed some light on the cause? Thanks.
This is because docker for aws uses a shell container when you ssh with the docker user (https://docs.docker.com/docker-for-aws/faqs/#what-are-the-editions-containers-running-after-deployment), so you don't actually have access to the host where the root dir is.
You can get access to the host through a privileged container, by first sshing with the docker user, then running something like:
docker run --rm -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh