【Docker】docker Overlay2 文件系统原理

大家好，欢迎来到IT知识分享网。

概述

overlayFS是被称为联合文件系统的其中一个解决方案。在2014年，发布了第一个版本并且合并到了Linux的内核3.18版本中，此时，在docker被称为是overlay文件驱动。后来在Linux 内核4.0 版本中进行了改进，称为overlay2。（overlay存在诸多性能和不稳定的问题，不推荐使用overlay，直接使用默认的overlay2即可）

overlay2工作原理

• lowerdir对应底层文件系统，是能被上层文件系统upperdir所共享的只读层
• workdir则可以理解为overlay2运作的一个工作目录，用于完成copy-on-write等操作
• overlay2运作时（也就是容器启动时），会将lowerdir、upperdir和workdir联合挂载到merged目录，为使用者提供一个“统一视图”

在具体使用中。可以查看 /var/lib/docker/overlay2/容器ID目录结构，并使用mount | grep overlay查看overlay的挂载情况。如下，确实此目录下的link、lower、work、diff等目录是通过挂载多个目录后，合并显示在一起的。

[root@k8s-m2 ~]# ls /var/lib/docker/overlay2/ 0ba0b18637ab4dd50ad1f3d5a62eebe0fd9c0 0ba0b18637ab4dd50ad1f3d5a62eebe0fd9c0-init ...... [root@k8s-m2 ~]# mount |grep overlay|more overlay on /var/lib/docker/overlay2/0ba0b18637ab4dd50ad1f3d5a62eebe0fd9c0/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SG5 OKVILBSA6FA4TG6QC3RW4CC:/var/lib/docker/overlay2/l/3SHYUCXLHK7TDFYIIMBGZMHY6J:/var/lib/docker/overlay2/l/67UXAPNT4CKASMUI7FGMJS2UOE:/var/lib/docker/overlay2/l/XDCHJ3X76XZLAA 43KZOHSUO5ZP,upperdir=/var/lib/docker/overlay2/0ba0b18637ab4dd50ad1f3d5a62eebe0fd9c0/diff,workdir=/var/lib/docker/overlay2/0ba0b18637a83260 08b4dd50ad1f3d5a62eebe0fd9c0/work) ...... 

overlay2 是如何存储文件的？

[root@kube1 ~]# sudo yum install -y yum-utils device-mapper-persistent-data lvm2 [root@kube1 ~]# yum install yum-utils -y [root@kube1 ~]# sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [root@kube1 ~]# sudo yum makecache fast [root@kube1 ~]# yum install -y docker-ce [root@kube1 ~]# systemctl start docker [root@kube1 overlay2]# ll total 0 brw------- 1 root root 253, 0 Jan 27 22:04 backingFsBlockDev drwx------ 2 root root 6 Jan 27 22:04 l 

[root@kube1 overlay2]# docker pull nginx Using default tag: latest latest: Pulling from library/nginx 2f44b7a888fa: Pull complete 8b7dd3ed1dc3: Pull complete 35497dd96569: Pull complete 36664b6ce66b: Pull complete 2df76c: Pull complete dc9c4fdb83d6: Pull complete 8056d2bcf3b6: Pull complete Digest: sha256:4c0fdaa8b6341bfdeca5f18fc80cff90527ee35efe1c327beac Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest 

[root@kube1 overlay2]# ls 10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503 backingFsBlockDev 2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1 c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0 410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620 48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2 l ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894 

4、首先来查看一下l目录，可以看到l目录是一堆软连接，把一些较短的随机串软连到镜像层的 diff 文件夹下，这样做是为了避免达到mount命令参数的长度限制

[root@kube1 overlay2]# ls -al l/ total 4 drwx------ 2 root root 244 Jan 27 22:12 . drwx--x--- 10 root root 4096 Jan 27 22:12 .. lrwxrwxrwx 1 root root 72 Jan 27 22:12 D6T76WE7CR4JMMPUM4NTO55AXG -> ../48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 GTRRPSN3NBPKHCIMECN7X7BJEP -> ../10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 JMSMKE6MVY4KJ5CB5ZDDIDVPW2 -> ../ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 LHNRUYWVMA3S5JGMBYRKW4BWHI -> ../c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 SX6PROXGAB4ZJPH7HTPYRQUMH2 -> ../cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 TMLMOWKXKUJ4WAWMNDCJVVLMBG -> ../2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/diff lrwxrwxrwx 1 root root 72 Jan 27 22:12 WCBZTXQX2YLYEZWCUYGFHKTAST -> ../410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f/diff 

5、docker image inspect nginx 查看nginx镜像的信息，每个镜像都会有一个GraphDriver.Data信息，这个信息指示了镜像是怎么存的

"GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2/diff:/var/lib/docker/overlay2/cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620/diff:/var/lib/docker/overlay2/10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503/diff:/var/lib/docker/overlay2/410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f/diff:/var/lib/docker/overlay2/ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff:/var/lib/docker/overlay2/c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0/diff", "MergedDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/merged", "UpperDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/diff", "WorkDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:571ade696b261f0ff46e3cdac4635afc009c4edcb95cd7e5f70ba0a07", "sha256:b6c2a8d6f0ac89ef77ef3d9d0dc5dfe0a5f20042e0afc0adeb", "sha256:b61d4b2cd2daf0c5876a35338c2beb5ae3f6bef479d25f05772a6a482", "sha256:eddcd06e5ef9bf6c55fa01a7d6963c435d5cf2bfb488d91aaa72d4a8", "sha256:b4adf0a8020bbfe6077e78fca48da4d77aa3ad2a72", "sha256:fbcc9bc44d3e165e7e4f56fb189a05ea5c562aec00d5e3fad309eb63cc", "sha256:009507beab5126f6363cb2badf370c9e95d4648c43e771f" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" } 

}

6、将这7个文件夹全部展开，可以看到目录结构几乎都是一致的，需要重点关注的是diff文件夹和lower文件。

[root@kube1 overlay2]# ls */ 10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503/: committed diff link lower work 2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/: diff link lower work 410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f/: committed diff link lower work 48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2/: committed diff link lower work ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/: committed diff link lower work c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0/: committed diff link cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620/: committed diff link lower work l/: D6T76WE7CR4JMMPUM4NTO55AXG JMSMKE6MVY4KJ5CB5ZDDIDVPW2 SX6PROXGAB4ZJPH7HTPYRQUMH2 WCBZTXQX2YLYEZWCUYGFHKTAST GTRRPSN3NBPKHCIMECN7X7BJEP LHNRUYWVMA3S5JGMBYRKW4BWHI TMLMOWKXKUJ4WAWMNDCJVVLMBG 

可以看到c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0文件夹中不存在lower文件，说明它是最底层的，相当于是根镜像，即docker pull时下载的第一层。
同时，下层diff文件夹下的文件，正是Linux文件目录结构。说明在nginx的dockerfile中，肯定有FROM 的操作。

实例说明：Dockerfile的每一个命令都可能引起了系统的变化，它的每一个变化都会记录一层diff文件。

容器怎么存储的？

1、当前环境只有一个nginx镜像，没有容器，/var/lib/docker/overlay2/ 目录下只有镜像层的存储目录。

2、docker run 启动一个容器

[root@kube1 overlay2]# docker run -id nginx 76cd5f0f5a3f8969d3061dfcf4ae4e9cce87 [root@kube1 overlay2]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 76cd5f0f5a3f nginx "/docker-entrypoint.…" 3 seconds ago Up 2 seconds 80/tcp festive_lederberg [root@kube1 overlay2]# ll total 0 drwx--x--- 4 root root 72 Jan 27 22:12 10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503 drwx--x--- 4 root root 72 Jan 28 01:10 2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1 drwx--x--- 4 root root 72 Jan 27 22:12 410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f drwx--x--- 4 root root 72 Jan 27 22:12 48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2 drwx--x--- 4 root root 72 Jan 27 22:12 ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894 brw------- 1 root root 253, 0 Jan 27 22:04 backingFsBlockDev drwx--x--- 3 root root 47 Jan 27 22:12 c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0 drwx--x--- 5 root root 69 Jan 28 01:10 cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb drwx--x--- 4 root root 72 Jan 28 01:10 cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb-init drwx--x--- 4 root root 72 Jan 27 22:12 cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620 

3、查看/var/lib/docker/overlay2/ 目录，发现新增了两个目录：其中带-init的目录是只读的；没有init的容器目录才是容器的读写目录

4、link和lower文件与镜像层的功能一致，link文件内容为该容器层的短 ID，lower文件为该层的所有父层镜像的短 ID。diff目录为容器的读写层，容器内修改的文件都会在diff中出现，merged目录为分层文件联合挂载后的结果，也是容器内的工作目录。

5、根据 docker inspect nginx获取到的GraphDriver.Data数据显示，merged目录，是lowerDir各个目录合并UpperDir各个目录后的结果。

[root@kube1 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 76cd5f0f5a3f nginx "/docker-entrypoint.…" 26 minutes ago Up 26 minutes 80/tcp festive_lederberg [root@kube1 ~]# docker inspect 76 "GraphDriver": { 
    "Data": { 
    "LowerDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb-init/diff:/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2eceade2458efe8721e2ec0007d7b1/diff:/var/lib/docker/overlay2/48305dd54e0eea026fe2c485c7d5b7c5939e5a062cc36bd03b2/diff:/var/lib/docker/overlay2/cb9d1b97aade9bd3e3a56268aa86a7ac0d0ade9bbb2b0679f4c0ed620/diff:/var/lib/docker/overlay2/10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912dafd7835ff83503/diff:/var/lib/docker/overlay2/410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f/diff:/var/lib/docker/overlay2/ada7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff:/var/lib/docker/overlay2/c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57ca0/diff", "MergedDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb/merged", "UpperDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb/diff", "WorkDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb/work" }, "Name": "overlay2" 

6、当我们进入容器创建一个文件，然后再宿主机上相应的merged目录也会出现这个文件。

[root@kube1 ~]# docker exec -it 76 /bin/bash root@76cd5f0f5a3f:/# ls bin boot dev docker-entrypoint.d docker-entrypoint.sh etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var root@76cd5f0f5a3f:/# touch a.txt root@76cd5f0f5a3f:/#  exit [root@kube1 merged]# pwd /var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981ccef7a3b375d7d9134cc3db932fd94b9bcb/merged [root@kube1 merged]# ls a.txt  

[root@k8s-m3 ~]# du -s /var/lib/docker/overlay2/*/merged | sort -rn | head  /var/lib/docker/overlay2/eacbb4f09241fabeffbdfb80ec8e12cbae3ef84e48/merged  /var/lib/docker/overlay2/2aaa6391bc4bdaa3380eea523c4940fa9ef677b81c3933d59d6/merged  /var/lib/docker/overlay2/10a01f4c8eda3b5a397a3f883bb4d7faccc0eb5f4b2fad4/merged  /var/lib/docker/overlay2/259b5e08903acc921fcd6f84d7ba95bdf8a63c0543d41c/merged  /var/lib/docker/overlay2/c48167ccceed1301bcb25ed4c1888e0dfbafd1bed86a7/merged 47656 /var/lib/docker/overlay2/fa40f1bbde2573b77f31e21d8392a0be9f9243afbb0e93ce0fdc392ab5/merged 47656 /var/lib/docker/overlay2/f8dcd1cce200d1c018f597dd55d75e42ce2526d9aa6a6e380ab4a7e285/merged 47656 /var/lib/docker/overlay2/c2dd77d6f03c259de3a03e8fefe4cf363ee1561bd0e5587cd5d3/merged 47656 /var/lib/docker/overlay2/4b0c49ba53bd03c2a4c81cd6ad899dfdcf859e115f6f6f204c4437b34b/merged 47656 /var/lib/docker/overlay2/43db1fd015f0f14b592f10f93d036fa2bde383c3f8da0a77acfc93f2faaac76d/merged [root@k8s-m3 ~]# docker ps -q | xargs docker inspect --format '{ 
   {.State.Pid}}, { 
   {.Id}}, { 
   {.Name}}, { 
   {.GraphDriver.Data.WorkDir}}' | grep eacbb4f09241fabeffbdfb80ec8e12cbae3ef84e48 15110, 808f1771db3776cd1bdf90702cd5470fc70d760d365ee56390c4f9ec, /k8s_nginx-gateway_my-nginx-7ff446c4f4-79wzx_default_2b11ff88-7c7b-4c73-b207-cb0b1d79924f_0, /var/lib/docker/overlay2/eacbb4f09241fabeffbdfb80ec8e12cbae3ef84e48/work 

结论： overlay2将镜像层和容器层都放在单独的目录，并且有唯一 ID，每一层仅存储发生变化的文件，最终使用联合挂载技术将容器层和镜像层的所有文件统一挂载到容器中，使得容器中看到完整的系统文件。

更多关于docker的知识分享，请前往博客主页。编写过程中，难免出现差错，敬请指出