1. IT分享知识网首页
  2. 未分类

容器启动加速-nydus

老牧童 • • 未分类

容器启动加速-nydusNydus 是一个镜像加速器 主要通过容器启动时懒加载镜像提高启动速度 大镜像可以秒级启动 后续的读写通过 COW 方式

大家好，欢迎来到IT知识分享网。

容器启动加速-nydus

1. 概述

Nydus是一个镜像加速器，主要通过容器启动时懒加载镜像提高启动速度, 大镜像可以秒级启动，后续的读写通过COW方式。以下是nydus的几个标签:

  • Nydus是蚂蚁、阿里云、字节等共建的开源容器镜像加速项目
  • CNCF孵化的项目Dragonfly中的一个子项目
  • 新的镜像格式，不同于oci
  • 良好的生态，兼容containerd、k8s、harbor、buildkit – 从构建、存储、部署、到CI集成，有丰富的解决方案
  • erofs+fscache+cachefiles 缓存方案: 使用内核缓存和只读文件系统进行io加速
    • cachefiles: 获取文件缓存
    • fscache: 管理cachefiles的缓存内容
    • erofs: 将缓存以文件系统接口方式提供访问

用Nydus对容器进行加速，当前可用，可以使用(k8s+)buildkit构建镜像，使用(k8s+)containerd进行发布, 使用skaffold进行CI(但路途坎坷，该文档内容主要为这部分)；未来可期，使用fscache+erofs进行底层存储加速。

1.1. 使用nydus需要考虑的问题

如果要落地Nydus，需要考虑如下几个问题:

  • 当前k8s container runtime 如果是docker，则需要切换成containerd，切换需要注意
    • pod服务可能会中断
    • 监控日志，原来通过docker接口，现在需要通过containerd接口 – 这部分k8s可以屏蔽
  • CI如何处理原有的docker build方式和现在buildkit构建方式
  • containerd功能相对简单，要用它来进行构建、运行等操作需要多个组件配合才行，要使用起来需要做不少操作:
    • containerd
    • nerdctl
    • buildkitd
    • buildctl …
    • nydus-snapshotter 要使用nydus镜像的话需要
  • ⚠️性能问题，写时复制从远端拉取，例如从镜像仓库拉取的话需要考虑
    • 镜像仓库本身的IO性能
    • 从pod到仓库的网络带宽
  • ⚠️容器化使用nydus-snapshotter有待改进
    现在nydus-snapshotter容器故障之后由于和容器启动需要nydus-snapshotter这样的循环依赖，所以无法自愈，需要
    • 删除containerd中的nydus-snapshotter配置
    • 重启containerd
    • 重新配置containerd

1.2. 当前实践探索中遇到的问题

在实践探索中主要有如下问题:

  • 涉及项目多，各项目版本不明确，遇到较多版本兼容问题
  • 文档分散，不易查阅
  • 实践类文档较少
  • 文档落后代码较多
  • 部分项目CI部分代码落后于主体代码

正在解决中的问题

  • 构建出来的本地镜像缺少layer无法run

2. 容器集成nydus

2.1. contanierd 集成【核心】

2.1.1. 版本说明

组件 版本 备注
内核 5.18.2-1.el7.elrepo.x86_64
containerd containerd.io 1.6.6 10ce7c7c9b6e0ea9b0c02b01407d3ae1
buildkit github.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1cbffbf0a43bea92215fbe.m 从https://github.com/nydusaccelerator/buildkit.git构建
nydus-image v2.2.1
containerd-nydus-grpc v0.8.0
nerdctl 1.3.1

2.1.2. 概念说明

  • nydus镜像: nydus格式的镜像，不同于现在的ociv1 ociv2，在使用时可进行懒加载，加快容器启动速度
  • nydus镜像构建
    • buildkitd: 构建nydus镜像的工具，需要增加nydus兼容配置, 一般以守护进程运行，客户端通过 buildctl 命令通过sock文件或tcp进行与服务端连接，也支持daemonless
    • nydus-image: nydus镜像工具，可以集成到buildkit中构建nydus格式的镜像，也可以单独使用进行nydus镜像构建、检查、导出等
  • nydus镜像运行
    • containerd: 容器运行时
    • nydus-snapshotter: 是一个containerd插件，用来支持contanierd使用nydus格式的镜像1
/usr/bin/nydusd fuse --thread-num 10 --config /var/lib/containerd-nydus/config/chmadgaihap9ldb1eg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/833/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/833/mnt --apisock /var/lib/containerd-nydus/socket/chmadgaihap9ldb1eg/api.sock --log-level debug --log-file /var/lib/containerd-nydus/logs/chmadgaihap9ldb1eg/nydusd.log

架构图如下，这里用的是fuse文件系统, erofs的社区文档比较少，暂时先用fuse

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jMiUFnzf-00)(null)]

如果用erofs

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vg8EBgO4-40)(null)]

2.1.3. 构建

2.1.3.1. 构建 buildkitd

官方社区仓库是 moby/buildkit, 这里用的nydusaccelerator的项目，是因为当前bug/新功能会先在nydusaccelerator发布，所以暂时先用这个项目

git clone https://github.com/nydusaccelerator/buildkit.git cd buildkit go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd mv ./bin/buildkitd /usr/bin/ chmod +x /usr/bin/buildkitd
2.1.3.2. 安装nydus
wget https://github.com/dragonflyoss/image-service/releases/download/v2.2.1/nydus-static-v2.2.1-linux-amd64.tgz tar zxf nydus-static-v2.2.1-linux-amd64.tgz cd nydus-static sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr/bin
2.1.3.3. 启动buildkitd进程

buildkit 配置2

cat > /etc/buildkit/buildkitd.toml <<EOF debug = false [worker.oci] enabled = false [worker.containerd] enabled = true # 构建成功的之后的镜像存储位置 namespace = "default" address = "/run/containerd/containerd.sock" enabled = true platforms = [ "linux/amd64", "linux/arm64" ] gc = true # gckeepstorage sets storage limit for default gc profile, in MB. gckeepstorage = 9000 EOF 
cat > /etc/systemd/system/buildkitd.service << EOF [Unit] Description=BuildKit Documentation=https://github.com/moby/buildkit [Service] Environment="NYDUS_BUILDER=/usr/bin/nydus-image" ExecStart=/usr/bin/buildkitd --config /etc/buildkit/buildkitd.toml [Install] WantedBy=multi-user.target EOF 
systemctl enable buildkitd systemctl start buildkitd
2.1.3.4. 构建镜像

构建一个简单的hello world镜像

  • hello world程序
mkdir golang-helloworld

编辑 main.go

package main import "fmt" func main() { 
    fmt.Println("Hello, world!") }

构建可执行文件

go build -o hello main.go
  • Dockerfile

编辑dockerfile

FROM ubuntu:20.04 COPY hello / ENTRYPOINT [ "/hello" ]
  • 构建

这里会构建一个nydus格式的镜像 名称为reg.harbor.com/hello/hello-k8s:v0.0.4, push 为 true 则构建完成之后推送远端仓库

buildctl build --progress=plain \ --frontend=dockerfile.v0 \ --local context=. \ --local dockerfile=. \ --output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.4,compression=nydus,oci-mediatypes=true,push=true
  • 校验

这里我们构建出来了nydus镜像，可以校验一下

nydusify check --target reg.harbor.com/hello/hello-k8s:v0.0.4 INFO[2023-05-24T10:59:56+08:00] Parsing image reg.harbor.com/hello/hello-k8s:v0.0.4 INFO[2023-05-24T10:59:56+08:00] Dumping OCI and Nydus manifests to ./output ... INFO[2023-05-24T10:59:57+08:00] Verified Nydus image reg.harbor.com/hello/hello-k8s:v0.0.4

这里check的是远端镜像，一切符合预期,是nydus镜像3

cat output/nydus_manifest.json { 
    "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { 
    "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:b13bcaf21d0748cccb353bdd8429e541da489d1833c0c9ea7", "size": 6940 }, "layers": [ { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } ... { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47acc91c91c", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob-ids": "[\"e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a\",\"dcdb421faaf9881e63baaf412d1f05aa78e2d8f5bb78f1d5\",\"68fb6d505c4733f3cab9ed30dbf46056b36630be2dafe48634af1e01\",\"32a3c81c8fb5b12c68d74e01b29cd02463d8e85c0296ac3e30874f0f38412ec9\",\"89fe8a5a29e872cb9afbdeffebe67f42a0d3fdeca754b7a55b\",\"43f9d24cfd0900b5ccbcdf305df5383fd165c723ca553\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab\"]", "containerd.io/snapshot/nydus-bootstrap": "true", "containerd.io/snapshot/nydus-fs-version": "" } } ] }

本地镜像暂时无法使用,使用会报错(这里是部署了nydus snapshotter之后)

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4 --snapshotter=nydus 
INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found" key="extract--gvw0 sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a" FATA[0000] failed to extract layer sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd0c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found

这里应该应该是nydus+fuse的bug，解决方法为在构建完成之后把本地镜像删除掉, 使用时通过远端拉取

nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

本地镜像构建出来的manifest和远端一致，这里不能用是因为存储的时候少存储了snapshotter，具体问题待社区确认

2.1.4. 部署

2.1.4.1. 部署nydus-snapshotter
  • 安装 containerd-nydus-grpc
wget https://github.com/containerd/nydus-snapshotter/releases/download/v0.8.0/nydus-snapshotter-v0.8.0-x86_64.tgz tar zxf nydus-snapshotter-v0.8.0-x86_64.tgz install -D -m 755 nydus-snapshotter/containerd-nydus-grpc /usr/bin
  • nydus-fusedev 配置
# nydus blob 缓存 mkdir /opt/nydus/cache 
cat > /etc/nydus/nydusd-config.fusedev.json <<EOF { "device": { "backend": { "type": "registry", "config": { "scheme": "https", "skip_verify": true, "timeout": 5, "connect_timeout": 5, "retry_limit": 4 } }, "cache": { "type": "blobcache", "config": { "work_dir": "/opt/nydus/cache" } } }, "mode": "direct", "digest_validate": false, "iostats_files": false, "enable_xattr": true, "fs_prefetch": { "enable": true, "threads_count": 4 } } EOF
  • nydus snapshotter 主进程配置4
cat /etc/nydus/config.toml version = 1 address = "/run/containerd-nydus/containerd-nydus-grpc.sock" daemon_mode = "multiple" cleanup_on_close = false [system] enable = true address = "/run/containerd-nydus/system.sock" [system.debug] daemon_cpu_profile_duration_secs = 5 pprof_address = "" [daemon] nydusd_path = "/usr/bin/nydusd" nydusimage_path = "/usr/bin/nydus-image" # fusedev or fscache fs_driver = "fusedev" # How to process when daemon dies: "none", "restart" or "failover" recover_policy = "restart" # Specify a configuration file for nydusd nydusd_config = "/etc/nydus/nydusd-config.fusedev.json" # The fuse or fscache IO working threads started by nydusd threads_number = 10 [log] # Print logs to stdout rather than logging files log_to_stdout = false level = "debug" log_rotation_compress = true log_rotation_local_time = true log_rotation_max_age = 7 log_rotation_max_backups = 5 log_rotation_max_size = 1 [metrics] # Enable by assigning an address, empty indicates metrics server is disabled address = ":9110" [remote] convert_vpc_registry = false [remote.mirrors_config] dir = "" [remote.auth] enable_kubeconfig_keychain = false kubeconfig_path = "" enable_cri_keychain = false image_service_address = "" [snapshot] enable_nydus_overlayfs = false sync_remove = false [cache_manager] disable = false gc_period = "24h" cache_dir = "/opt/nydus/cache/" [image] public_key_file = "" validate_signature = false [experimental] enable_stargz = false enable_referrer_detect = false 
cat > /etc/systemd/system/nydus-snapshotter.service <<EOF [Unit] Description=nydus snapshotter After=network.target Before=containerd.service [Service] Type=simple Environment=HOME=/root ExecStart=/usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config.toml Restart=always RestartSec=1 KillMode=process OOMScoreAdjust=-999 StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target EOF 
systemctl daemon-reload systemctl enable nydus-snapshotter systemctl start nydus-snapshotter
2.1.4.2. 配置containerd支持nydus snapshotter

编辑containerd配置 /etc/containerd/config.toml

version = 2 state = "/run/containerd" [grpc] address = "/run/containerd/containerd.sock" uid = 0 gid = 0 max_recv_message_size =  max_send_message_size =  [timeouts] "io.containerd.timeout.shim.cleanup" = "5s" "io.containerd.timeout.shim.load" = "5s" "io.containerd.timeout.shim.shutdown" = "3s" "io.containerd.timeout.task.state" = "2s" [plugins] [plugins."io.containerd.grpc.v1.cri"] # k8s原来用的container runtime是docker，则需要配置pause镜像，否则runtime改编之后k8s原有pod无法启动 sandbox_image = "reg.harbor.com/kubernetes/pause:3.6" [plugins."io.containerd.grpc.v1.cri".cni] # cni插件路径 bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" max_conf_num = 1 conf_template = "" [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] # Nydus插件主要配置 [proxy_plugins] [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock" [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "nydus" disable_snapshot_annotations = false discard_unpacked_layers = false

修改完毕重启containerd

systemctl restart containerd

2.1.5. 使用nydus镜像启动containerd容器

containerd也有类似docker的命令行 nerdctl

  • 安装 nerdctl
wget https://github.com/containerd/nerdctl/releases/download/v1.3.1/nerdctl-1.3.1-linux-amd64.tar.gz tar zxvf nerdctl-1.3.1-linux-amd64.tar.gz install -D -m 755 nerdctl /usr/bin/
  • 配置 nerdctl

默认使用 nydus-snapshotter

cat > /etc/nerdctl/nerdctl.toml <<EOF debug = false debug_full = false address = "unix:///var/run/containerd/containerd.sock" namespace = "default" snapshotter = "nydus" experimental = true EOF
  • 使用构建阶段产生的镜像运行容器

先删除本地镜像 nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4 # 输出 reg.harbor.com/hello/hello-k8s:v0.0.4: resolved |++++++++++++++++++++++++++++++++++++++| manifest-sha256:c6ab4a56b80dbda42fc31b038d6f6989d8d9e801cb: exists |++++++++++++++++++++++++++++++++++++++| config-sha256:b13bcaf21d0748cccb353bdd8429e541da489d1833c0c9ea7: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47acc91c91c: done |++++++++++++++++++++++++++++++++++++++| elapsed: 0.6 s total: 2.2 Mi (3.6 MiB/s) Hello, world!

使用 nerdctl 命令时可能会遇到 failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]" cni不支持的问题，需要升级cni

https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-arm64-v1.1.1.tgz tar zxf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin

升级完毕即可用

到这里使用nydus加速containerd容器启动，基本完成。使用erofs待探索

2.2. kubernertes集成

2.2.1. 版本说明

组件 版本 备注
k8s v1.22.10
containerd containerd.io 1.6.6 10ce7c7c9b6e0ea9b0c02b01407d3ae1
buildkitd github.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1cbffbf0a43bea92215fbe.m 从https://github.com/nydusaccelerator/buildkit.git构建
nydus-snapshotter v0.6.1 通过helm部署5,生产用的话需要更新下

2.2.2. k8s集群准备

首先需要一个 container runtime为containerd的集群

kubectl describe nodes |grep -i runtime Container Runtime Version: docker://20.10.8 Container Runtime Version: docker://20.10.8 Container Runtime Version: docker://20.10.8

如果是docker，则需要更改为containerd

2.2.2.1. 将k8s cri从docker替换为containerd

此处操作比较粗糙，生产切换请慎重

  • 更改 kubelet 配置

编辑/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

增加/在KUBELET_EXTRA_ARGS追加如内容

Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
  • 创建 containerd 配置

新建 “

version = 2 state = "/run/containerd" [plugins] [plugins."io.containerd.grpc.v1.cri"] sandbox_image = "reg.harbor.com/kubernetes/pause:3.6" You have new mail in /var/spool/mail/root

重启containerd kubelet服务

systemctl daemon-reload systemctl restart containerd kubelet

如果 kubelet 启动失败，请查看日志做对应处理 journalctl -xu kubelet -f

启动成功之后检查如下输出之后说明切换成功

kubectl describe nodes |grep -i runtime # 三个节点 Container Runtime Version: containerd://1.6.6 Container Runtime Version: containerd://1.6.6 Container Runtime Version: containerd://1.6.6

2.2.3. 部署buildkit

2.2.3.1. 构建buildkit镜像

社区的镜像不支持nydus，需要构建镜像

git clone https://github.com/nydusaccelerator/buildkit.git cd buildkit

修改 Dockerfile

diff --git a/Dockerfile b/Dockerfile index b..b006e584  --- a/Dockerfile +++ b/Dockerfile @@ -97,7 +97,7 @@ ARG TARGETPLATFORM RUN --mount=target=. --mount=target=/root/.cache,type=cache \ --mount=target=/go/pkg/mod,type=cache \ --mount=source=/tmp/.ldflags,target=/tmp/.ldflags,from=buildkit-version \ - CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \ + CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo nydus static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \ xx-verify --static /usr/bin/buildkitd FROM scratch AS binaries-linux-helper @@ -176,6 +176,7 @@ ARG ROOTLESSKIT_VERSION RUN git clone https://github.com/rootless-containers/rootlesskit.git /go/src/github.com/rootless-containers/rootlesskit WORKDIR /go/src/github.com/rootless-containers/rootlesskit ARG TARGETPLATFORM +ENV GOPROXY "https://goproxy.cn,direct" RUN --mount=target=/root/.cache,type=cache \ git checkout -q "$ROOTLESSKIT_VERSION" && \ CGO_ENABLED=0 xx-go build -o /rootlesskit ./cmd/rootlesskit && \ @@ -203,6 +204,7 @@ RUN mkdir -p /out/nydus-static && tar xzvf nydus-static-$NYDUS_VERSION-$TARGETOS FROM buildkit-export AS buildkit-linux COPY --link --from=binaries / /usr/bin/ +COPY --from=nydus /out/nydus-static/* /usr/bin/ ENTRYPOINT ["buildkitd"] FROM binaries AS buildkit-darwin

主要有三处修改:

  • 编译增加nydus支持 – 必须
  • 增加goproxy配置 – 国内拉依赖包会失败，需要配置goproxy
  • 将nydus-image打入镜像，这步经测试可有可无
make build

会构建出两个镜像，我们只使用非rootless的 moby/buildkit:local

2.2.3.2. 按需修改yaml并部署
git clone https://github.com/nydusaccelerator/buildkit.git cd buildkit

这里使用deployment+service.privileged 6

编辑 deployment+service.privileged.yaml，去掉了tls配置，增加了NYDUS_BUILDER环境变量，最终如下

镜像 shoery/buildkit:76aeaa55

apiVersion: apps/v1 kind: Deployment metadata: labels: app: buildkitd name: buildkitd spec: replicas: 1 selector: matchLabels: app: buildkitd template: metadata: labels: app: buildkitd spec: containers: - name: buildkitd env: - name: NYDUS_BUILDER value: /usr/bin/nydus-image image: shoery/buildkit:76aeaa55 args: - --addr - unix:///run/buildkit/buildkitd.sock - --addr - tcp://0.0.0.0:1234 # the probe below will only work after Release v0.6.3 readinessProbe: exec: command: - buildctl - debug - workers initialDelaySeconds: 5 periodSeconds: 30 # the probe below will only work after Release v0.6.3 livenessProbe: exec: command: - buildctl - debug - workers initialDelaySeconds: 5 periodSeconds: 30 securityContext: privileged: true ports: - containerPort: 1234 --- apiVersion: v1 kind: Service metadata: labels: app: buildkitd name: buildkitd spec: ports: - port: 1234 protocol: TCP selector: app: buildkitd

部署

kubectl apply -f examples/kubernetes/deployment+service.privileged.yaml

构建镜像时仅需客户端有 buildctl 即可, 通过 --addr tcp://0.0.0.0:1234 连接到 buildkitd

2.2.4. 部署 nydus-snapshotter

run_nydus_in_kubernetes

这里注意用 v0.8.2 以上的镜像版本

这里没太多修改的地方, 两个地方需要修改，其他按文档部署即可

git clone https://github.com/dragonflyoss/helm-charts.git cd helm-charts
2.2.4.1. 更改nydus镜像

新建 config-nydus.yaml

name: nydus-snapshotter image: ghcr.io/containerd/nydus-snapshotter tag: v0.8.2 pullPolicy: IfNotPresent hostNetwork: true dragonfly: enable: false containerRuntime: containerd: enable: true
2.2.4.2. 修改configmap

如果不增加nydus.toml配置，懒加载不会生效

编辑 charts/nydus-snapshotter/templates/nydus-snapshotter/snapshotter-configmap.yaml

apiVersion: v1 kind: ConfigMap metadata: name: { 
   { 
    template "nydus-snapshotter.fullname" . }} labels: app: { 
   { 
    template "nydus-snapshotter.fullname" . }} chart: { 
   { 
    .Chart.Name }}-{ 
   { 
    .Chart.Version }} release: { 
   { 
    .Release.Name }} heritage: { 
   { 
    .Release.Service }} component: { 
   { 
    .Values.name }} data: config.json: |- { 
    "device": { 
    "backend": { 
    "type": "registry", "config": { 
    { 
   { 
   - if .Values.dragonfly.enable }} "mirrors": { 
   { 
    mustToJson .Values.dragonfly.mirrorConfig }}, { 
   { 
   - end }} "timeout": 5, "connect_timeout": 5, "retry_limit": 2 } }, "cache": { 
    "type": "blobcache", "config": { 
    "work_dir": "/var/lib/nydus/cache/" } } }, "mode": "direct", "digest_validate": false, "iostats_files": false, "enable_xattr": true, "fs_prefetch": { 
    "enable": true, "threads_count": 8, "merging_size": , "prefetch_all": true } } nydus.toml: |- version = 1 address = "/run/containerd-nydus/containerd-nydus-grpc.sock" daemon_mode = "multiple" cleanup_on_close = false [system] enable = true address = "/run/containerd-nydus/system.sock" [system.debug] daemon_cpu_profile_duration_secs = 5 pprof_address = "" [daemon] nydusd_path = "/usr/local/bin/nydusd" nydusimage_path = "/usr/local/bin/nydus-image" fs_driver = "fusedev" recover_policy = "restart" nydusd_config = "/etc/nydus/config.json" threads_number = 10 [log] log_to_stdout = false level = "debug" log_rotation_compress = true log_rotation_local_time = true log_rotation_max_age = 7 log_rotation_max_backups = 5 log_rotation_max_size = 1 [metrics] address = ":9110" [remote] convert_vpc_registry = false [remote.mirrors_config] dir = "" [remote.auth] enable_kubeconfig_keychain = false kubeconfig_path = "" enable_cri_keychain = false image_service_address = "" [snapshot] enable_nydus_overlayfs = false sync_remove = false [cache_manager] disable = false gc_period = "24h" [image] public_key_file = "" validate_signature = false [experimental] enable_stargz = false enable_referrer_detect = false

安装

helm install --wait --timeout 10m --dependency-update \ --create-namespace --namespace nydus-system \ -f config-nydus.yaml \ nydus-snapshotter charts/nydus-snapshotter
2.2.4.3. 重启nydus-snapshotter

后续维护中可能会有重启 nydus-snapshotter 的需求

由于修改containerd配置使用nyduys snapshotter之后, containerd和snapshotter会互相依赖，重启nydus-snapshotter容器会起不来，所以，每次重启都先把 /etc/containerd/config.toml 中如下部分去掉

[plugins."io.containerd.grpc.v1.cri".containerd] discard_unpacked_layers = false disable_snapshot_annotations = false snapshotter = "nydus" [proxy_plugins] [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd
重新创建 nydus-snapshotter pod

2.2.5. 验证

验证点: 在pod中构建nydus格式的镜像，使用pod启动nydus镜像容器

2.2.5.1. 在pod中构建nydus格式的镜像

要挂载 /root/.docker, 否则push镜像可能会没权限

挂载golang-helloworld的目的是将 hello 和 Dockerfile挂载进去

创建 build-nydus-image.yaml

apiVersion: v1 kind: Pod metadata: name: build-pod spec: restartPolicy: Never containers: - name: build-container image: buildkit:local command: ["buildctl", "--addr", "tcp://buildkitd.default:1234", "build", "--progress=plain", "--frontend=dockerfile.v0", "--local", "context=.", "--local", "dockerfile=.", "--output", "type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true"] volumeMounts: - name: host-path-volume mountPath: /opt/golang-helloworld - name: docker-config-volume mountPath: /root/.docker workingDir: /opt/golang-helloworld volumes: - name: host-path-volume hostPath: path: /home/elrond/nydus/golang-helloworld - name: docker-config-volume hostPath: path: /root/.docker

命令说明

buildctl \ # buildkitd的svc:port --addr tcp://default.buildkitd:1234 \ build \ # 输出的详细程度 --progress=plain \ # build frontend 类型 --frontend=dockerfile.v0 \ # 指定本地context路径 --local context=. \ # 制定本地dockerfile路径 --local dockerfile=. \ # 指定dockerfile文件，如果为空，默认值为Dockerfile --opt filename=hello.dockerfile # 镜像tag与类型、是否推送到远端 --output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true

开始构建

kubectl apply -f build-nydus-image.yaml

构建成功

k get po NAME READY STATUS RESTARTS AGE build-pod 0/1 Completed 0 23s
2.2.5.2. 使用pod启动nydus镜像容器

创建 nydus-pod.yaml

apiVersion: v1 kind: Pod metadata: name: nydus-pod spec: containers: - name: hello image: reg.harbor.com/hello/hello-k8s:v0.0.6 imagePullPolicy: Always command: ["sh", "-c"] args: - tail -f /dev/null

运行

kubectl apply -f nydus-pod.yaml

这里可能会因为构建的镜像不正常，导致pod起不来，可以使用标准镜像 ghcr.io/dragonflyoss/image-service/nginx:nydus-latest 做测试

到这里在k8s上完成了nydus镜像的构建，和nydus镜像格式容器的启动

2.3. docker 集成

参照docker-env-setup

docker-nydus-graphdriver 尚在 Experimental 状态，且k8s在转向containerd，云原生社区对docker支持变弱，所以后续不考虑使用docker

3. CI集成 – skaffold集成nydus

编辑 skaffold.yaml

apiVersion: skaffold/v4beta4 kind: Config build: tagPolicy: envTemplate: template: "{ 
   {.FOO}}" artifacts: - image: reg.harbor.com/hello/hello-k8s custom: buildCommand: buildctl --addr tcp://10.233.98.169:1234 build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=reg.harbor.com/hello/hello-k8s:"{ 
   {.FOO}}",compression=nydus,oci-mediatypes=true,push=true --opt filename=hello.dockerfile local: push: true

入参:

  • –addr tcp://10.233.98.169🔢 容器化的话，这个是svc的host和port
  • –local dockerfile=.: dockerfile路径
  • –output name=reg.harbor.com/hello/hello-k8s:“{
    {.FOO}}”: 镜像tag
  • –opt filename: dockerfile文件名

其他都是默认值

4. nydus镜像启动pod与ociv1镜像启动pod效果对比

镜像 ociv1拉取时间 nydus拉取时间
wordpress 12.s 1.s

4.1. wordpress镜像

先把镜像拉下来,推送到自己的仓库，然后转化一下推送到自己的仓库，用镜像启动k8s pod

docker pull wordpress:latest docker tag wordpress:latest reg.harbor.com/hello/wordpress:latest nydusify -D convert --nydus-image /usr/local/bin/nydus-image \ --source reg.harbor.com/hello/wordpress:latest \ --target reg.harbor.com/hello/wordpress:latest-nydus

4.1.1. oci v1镜像容器pulling image时间

新建 wordpress-ociv1.yaml

apiVersion: v1 kind: Pod metadata: name: wordpress-ociv1 spec: containers: - name: wordpress image: reg.harbor.com/hello/wordpress:latest imagePullPolicy: Always command: - sleep - "" resources: limits: cpu: "0.5" memory: "256Mi" 
kubectl apply -f wordpress-ociv1.yaml kubectl describe po wordpress-ociv1 
Normal Pulling 3m20s kubelet Pulling image "reg.harbor.com/hello/wordpress:latest" Normal Pulled 3m7s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest" in 12.s

4.1.2. nydus镜像容器pulling image时间

新建 wordpress-nydus.yaml

apiVersion: v1 kind: Pod metadata: name: wordpress-nydus spec: containers: - name: wordpress image: reg.harbor.com/hello/wordpress:latest-nydus imagePullPolicy: Always command: - sleep - "" resources: limits: cpu: "0.5" memory: "256Mi" 
Normal Pulling 40s kubelet Pulling image "reg.harbor.com/hello/wordpress:latest-nydus" Normal Pulled 39s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus" in 1.s

5. 问题记录

5.1. 镜像无法构建

git clone https://github.com/nydusaccelerator/buildkit.git cd buildkit go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd mv ./bin/buildkitd /usr/bin/ chmod +x /usr/bin/buildkitd
  • 原生的buildkit不支持nydus格式构建，需要重新编译支持
  • 官方的buildkit对大镜像构建有bug，需要用nydus官方上游的仓库

5.2. containerd 命令速记

container与task7

主要用nerdctl工具

# 启动容器 ctr run -d docker.io/library/nginx:alpine my-container # 容器列表 ctr -n default c ls # task列表 ctr -n default task ls # 删除容器 ctr -n default delete xxxxx # 容器交互式命令 ctr task exec --tty --exec-id my-container my-container bash # 镜像列表 ctr -n default images ls # 获取contanierd镜像的manifest 先找到RepoDigests sha256 nerdctl inspect d88ba62a3cbb cat /opt/docker/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbbcbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3 # 转换镜像并推送到仓库 nydusify -D convert --nydus-image /usr/local/bin/nydus-image --source reg.harbor.com:443/library/golang@sha256:4b5cc6ca083856af8f58ab332f279b4401e1e46cc6dd5f88c563df24a7 --target reg.harbor.com:443/library/golang:1.18.7-nydus nydusify -D convert --nydus-image /usr/local/bin/nydus-image --source ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 --target reg.harbor.com/hello/ubuntu:nydus-nightly-v6

5.3. 镜像无法下载

nerdctl pull ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:cb13d8b2c3b275fdb610e31462ab5dc588d4735b0c469ff867f86f617f: exists |++++++++++++++++++++++++++++++++++++++| manifest-sha256:d9ea5e7108f5405da73aeda2ab976dac4c06e07a473ebb86c3: exists |++++++++++++++++++++++++++++++++++++++| config-sha256:d89d923e5b5aa9c9e22ec60bcdfaa02d24106b525f6171e: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:f264de8a417d8927c431ef2a74286e24d716c3447a96ef7c7bd95d3af4ab3331: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:c279d170bbf5c24d3b0b80bc1b49dd0f9f3f62cb010e98295b8d0e1f7c: downloading |+++++++++++++++++++++++++++++++++++++-| 31.0 MiB/31.1 MiB docker.io/library/alpine:latest: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:02bb6ffbc2809c5d1b41eab5afba33cb1af4444c9b11: done |++++++++++++++++++++++++++++++++++++++| manifest-sha256:c0669ef34cdc14332c0f1ab0c2c01acb91d96014b172f1a76f3a39e63d1f0bda: done |++++++++++++++++++++++++++++++++++++++| config-sha256:5e2b554c1c45d22c9d1aae320a26011b76c08631ac896cbc3625e3e: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0fecc07637e94dfd1d0639b6: done |++++++++++++++++++++++++++++++++++++++| elapsed: 44.9s total: 3.2 Mi (73.9 KiB/s) FATA[0045] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-05-15T21:27:17+08:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]" Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/286be7aef32271dac13b13e0a0060f9aa9c/oci-hook.createRuntime.log: file already closed: unknown

解决 升级cni到v1.1.1

5.4. 如何判断一个镜像是否是nydus镜像

如何判断一个镜像是否是nydus镜像8

5.4.1. 通过本地manifest判断

如果镜像在containerd上

nerdctl inspect d88ba62a3cbb # 找到RepoDigests sha256 d88ba62a3cbbcbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3 
[ { 
    "Id": "sha256:13afb147bdb4382b43a833bdca8bc6fca92cc2a60b02e4fc718b356d97", "RepoTags": [ "reg.harbor.com/hello/helloworld:v0.0.1" ], "RepoDigests": [ "reg.harbor.com/hello/helloworld@sha256:d88ba62a3cbbcbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3" ], "Comment": "buildkit.dockerfile.v0", "Created": "2023-05-16T20:28:18.+08:00", "Author": "", "Config": { 
    "AttachStdin": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Entrypoint": [ "/hello" ], "Labels": { 
    "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04" } }, "Architecture": "amd64", "Os": "linux", "Size": , "RootFS": { 
    "Type": "layers", "Layers": [ "sha256:f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca", "sha256:95fc9a6f9c1a1ef74f6545d70cf0cd8793aa134f4b1f51006", "sha256:6f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24", "sha256:ed593dc0cd5ea3a5b20bfb2c73f6108badffb30b080de9edcd" ] }, "Metadata": { 
    "LastTagTime": "0001-01-01T00:00:00Z" } } ]

查看manifest

# containerd的默认是/var/lib/containerd cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbbcbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

mediaType是nydus就是nydus镜像

{ 
    "mediaType": "application/vnd.oci.image.manifest.v1+json", "schemaVersion": 2, "config": { 
    "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:13afb147bdb4382b43a833bdca8bc6fca92cc2a60b02e4fc718b356d97", "size": 2137 }, "layers": [ { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:95fc9a6f9c1a1ef74f6545d70cf0cd8793aa134f4b1f51006", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:6f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:bb55b253c36e5a008051aad37f2554a65eca08666d76e5ee1b84dbf4c", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca\",\"95fc9a6f9c1a1ef74f6545d70cf0cd8793aa134f4b1f51006\",\"6f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24\"]", "containerd.io/snapshot/nydus-bootstrap": "true", "containerd.io/snapshot/nydus-fs-version": "" } } ] }

5.4.2. 通过nydusify check检查

nydusify check --target reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5 
INFO[2023-05-18T10:46:09+08:00] Parsing image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5 INFO[2023-05-18T10:46:11+08:00] Dumping OCI and Nydus manifests to ./output INFO[2023-05-18T10:46:11+08:00] Pulling Nydus bootstrap to output/nydus_bootstrap INFO[2023-05-18T10:46:11+08:00] Checking Nydus manifest INFO[2023-05-18T10:46:11+08:00] Checking Nydus bootstrap RAFS filesystem metadata is valid, referenced data blobs: 0: f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca, compressed data size 0x1e5beec, compressed file size 0x1e5c0ec, uncompressed file size 0x49b715a, chunks: 0x9d4, features: cap_toc tar-header 1: ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129, compressed data size 0x459da, compressed file size 0x45bda, uncompressed file size 0xb3c90, chunks: 0x1, features: cap_toc tar-header 2: 5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab, compressed data size 0x, compressed file size 0x, uncompressed file size 0x1ad499, chunks: 0x2, features: cap_toc tar-header INFO[2023-05-18T10:46:11+08:00] Verified Nydus image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

查看nydus manifest

cat output/nydus_manifest.json 
{ 
    "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { 
    "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324", "size": 2137 }, "layers": [ { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:ea9bc6d05ad80945a8d360d99a48b0d3ef070cf0cb6115", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca\",\"ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab\"]", "containerd.io/snapshot/nydus-bootstrap": "true", "containerd.io/snapshot/nydus-fs-version": "" } } ] }

5.5. nydus snapshotter无法启动

可能是有另一个 nydus-snapshotter进程在启动 ps -ef|grep containerd-nydus-grpc

/usr/local/bin/containerd-nydus-grpc --root /opt/docker/io.containerd.snapshotter.v1.nydus --config-path /etc/nydus/nydusd-config.fusedev.json --config /etc/nydus/config.toml --log-to-stdout --log-level trace INFO[2023-05-17T15:08:33.0+08:00] Start nydus-snapshotter. PID 25641 Version v0.8.0 FsDriver fusedev DaemonMode multiple FATA[2023-05-17T15:08:37.0+08:00] failed to start nydus-snapshotter error="failed to initialize snapshotter: create database: timeout"ß

5.6. buildkit构建nydus镜像构建不成功

buildctl build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=reg.harbor.com/hello/helloworld:v0.0.4-nydus-from-golang,compression=nydus,oci-mediatypes=true,push=true 
#6 [2/2] COPY hello / #6 ERROR: mount callback failed on /run/user/0/containerd-mount: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length  bytes ------ > [2/2] COPY hello /: ------ Dockerfile:2 -------------------- 1 | FROM reg.harbor.com/library/golang:1.18.7-nydus 2 | >>> COPY hello / 3 | ENTRYPOINT [ "/hello" ] 4 | -------------------- error: failed to solve: OutOfRange: failed to compute cache key: mount callback failed on /run/user/0/containerd-mount: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length  bytes

nydus-image log

time="2023-05-17T15:34:24+08:00" level=debug msg="saved dapdgw7aeqou4bbcymhfxc94n as context:context:" span="[internal] load build context" spanID=7108c9f5c020f1dd traceID=3841af58efa9339c11d3c4107ad1e65a time="2023-05-17T15:34:24+08:00" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown" failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown 20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml github.com/moby/buildkit/solver.(*edge).createInputRequests.func1.1 /root/buildkit/solver/edge.go:839 github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2 /root/buildkit/solver/internal/pipe/pipe.go:82 runtime.goexit /root/sdk/go1.18/src/runtime/asm_amd64.s:1571 20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml main.unaryInterceptor.func1 /root/buildkit/cmd/buildkitd/main.go:576 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1 /root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1 /root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 github.com/moby/buildkit/api/services/control._Control_Solve_Handler /root/buildkit/api/services/control/control.pb.go:2440 google.golang.org/grpc.(*Server).processUnaryRPC /root/buildkit/vendor/google.golang.org/grpc/server.go:1340 google.golang.org/grpc.(*Server).handleStream /root/buildkit/vendor/google.golang.org/grpc/server.go:1713 google.golang.org/grpc.(*Server).serveStreams.func1.2 /root/buildkit/vendor/google.golang.org/grpc/server.go:965 runtime.goexit /root/sdk/go1.18/src/runtime/asm_amd64.s:1571

使用 nydusaccelerator/buildkit 构建buildkit之后未复现

5.7. dragonflyoss仓库的镜像无法 inspect

 nerdctl inspect ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xda8eb3] goroutine 1 [running]: github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat.ImageFromNative(0xc0006cc000) /home/runner/work/nerdctl/nerdctl/pkg/inspecttypes/dockercompat/dockercompat.go:297 +0x253 github.com/containerd/nerdctl/pkg/cmd/image.Inspect.func1({0x13f0190?, _}, { 
  { 
  {0xc0006b0280, 0x3a}, 0x0, { 
  {0xc0006ae0c0, 0x27}, {0xc0006c0050, 0x47}, 0x248, ...}, ...}, ...}) /home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:52 +0x205 github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).Walk(0xc000, {0x13f0190, 0xc00014cd20}, {0x7ffc1ceb0732, 0x3a}) /home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:81 +0x666 github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).WalkAll(0x0?, {0x13f0190, 0xc00014cd20}, {0xc0005477c0?, 0x1, 0x103b5e0?}, 0x1) /home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:98 +0xea github.com/containerd/nerdctl/pkg/cmd/image.Inspect({_, _}, _, {_, _, _}, { 
  {0x13e7d20, 0xc00012e008}, {0x0, 0x0, ...}, ...}) /home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:64 +0x1a6 main.inspectAction(0xc000, {0xc00034e040, 0x1, 0x1?}) /home/runner/work/nerdctl/nerdctl/cmd/nerdctl/inspect.go:147 +0xb7c github.com/spf13/cobra.(*Command).execute(0xc000, {0xc000, 0x1, 0x1}) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 github.com/spf13/cobra.(*Command).ExecuteC(0xc000004300) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 main.xmain() /home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:136 +0x9f main.main() /home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:119 +0x19

原因未知

5.8. nydus镜像与oci镜像manifest的区别

  • ociv19
{ 
    "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { 
    "mediaType": "application/vnd.oci.image.config.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0e0d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "size": 32654, "digest": "sha256:dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0" }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdcd94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "size": 73109, "digest": "sha256:ec4b77945c89419d1af06b5f7636b4ac3da7fad" } ], "subject": { 
    "mediaType": "application/vnd.oci.image.manifest.v1+json", "size": 7682, "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a" }, "annotations": { 
    "com.example.key1": "value1", "com.example.key2": "value2" } }
  • docker v2.2
{ 
    "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4710, "digest": "sha256:016db0f74f4d70593d50a571e76792fe14fb791f9f6fbef", "platform": { 
    "architecture": "amd64", "os": "linux" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4708, "digest": "sha256:1a3cd94d36b750c10782c41bfcf9b5b81da4bb136dae7c11e439f", "platform": { 
    "architecture": "arm", "os": "linux", "variant": "v5" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4708, "digest": "sha256:fac5d849d932a3a33d421f2918d6b36214fb8f048a04533e95bba5a1da263dc5", "platform": { 
    "architecture": "arm", "os": "linux", "variant": "v7" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4709, "digest": "sha256:6196a365b5ac3c585d65aeb73b68810fc9a24453d5e78ae6a8f229a0d8edf060", "platform": { 
    "architecture": "arm64", "os": "linux", "variant": "v8" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4710, "digest": "sha256:d9cda8b6e3ae929ad93f217dabb75fc09bda99b46aa84", "platform": { 
    "architecture": "386", "os": "linux" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4708, "digest": "sha256:d4089c75201a6f692e55a9b5480ab7bb94a58937d4452d444f0d1ccd5", "platform": { 
    "architecture": "mips64le", "os": "linux" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4710, "digest": "sha256:bc495de2c53f8e0df08ee58ab06ef2e4faef649c8285c27a3ee9674eb", "platform": { 
    "architecture": "ppc64le", "os": "linux" } }, { 
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 4708, "digest": "sha256:bb98ee8ed8c2bf30b82ad9218cde446a7ea342d58b1826b2b29aeccaad", "platform": { 
    "architecture": "s390x", "os": "linux" } } ] }
  • nydus
{ 
    "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { 
    "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324", "size": 2137 }, "layers": [ { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1", "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob": "true" } }, { 
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:ea9bc6d05ad80945a8d360d99a48b0d3ef070cf0cb6115", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cffb0fbfaa58afdf0cba82cca\",\"ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454bc6daaab\"]", "containerd.io/snapshot/nydus-bootstrap": "true", "containerd.io/snapshot/nydus-fs-version": "" } } ] }

5.9. 构建的镜像list时告警

nerdctl images WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.7-nydus-nightly-v5" for platform "linux/amd64" error="snapshot sha256:0ee114e6b61fec8478b90a426bdcbbc145d5857ddc8e09111f391 does not exist: not found" REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE reg.harbor.com/hello/helloworld v0.0.7-nydus-nightly-v5 484d07c284ad 33 seconds ago linux/amd64 0.0 B 345.5 MiB

本地镜像存储问题 删除本地镜像即可，具体问题尚未可知

5.10. 问题 buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题

  • 现象:
ctr run -d reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5 hello

错误输出

INFO[0000] apply failure, attempting cleanup error="failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown" key="extract--FeAw sha256:f5294afb011d1423a0b7c580b44338d6fb790eaf0ebdcc838fcb107d29" ctr: failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5ab7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

构建完使用 nerdctl命令查看有告警信息

 nerdctl images WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5" for platform "linux/amd64" error="snapshot sha256:66e2cbc88c92b00741c687d56bedd11654d45c08dec2fcbfb does not exist: not found" REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE xxxx/hello/helloworld v0.0.10-nydus-nightly-v5 b3faf19db1a6 11 minutes ago linux/amd64 0.0 B 32.6 MiB

containerd配置

[proxy_plugins] [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock" [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "nydus" disable_snapshot_annotations = false discard_unpacked_layers = false
  • 环境说明
    • containerd containerd.io 1.6.6 10ce7c7c9b6e0ea9b0c02b01407d3ae1
    • buildkitd -v buildkitd github.com/moby/buildkit v0.0.0+unknown 是通过ttps://github.com/nydusaccelerator/buildkit.git 构建的
    • nerdctl -v nerdctl version 1.3.1 在 /etc/nerdctl/nerdctl.toml 指定了 snapshotter
    debug = false debug_full = false address = "unix:///var/run/containerd/containerd.sock" namespace = "default" snapshotter = "nydus" experimental = true
    • buildkitd 启动命令
    NYDUS_BUILDER=/usr/bin/nydus-image buildkitd --oci-worker-snapshotter=nydus --config /etc/buildkit/buildkitd.toml`
    • 构建命令
    buildctl build --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=xxxx/helloworld:v0.0.10-nydus-nightly-v5,compression=nydus,oci-mediatypes=true

本地镜像存储问题 删除本地镜像即可，问题同上，nydus-snapshotter+fusedev的问题，具体尚未可知，删除本地镜像使用远端镜像即可

5.11. 运行oci-ref镜像导致nydusd进程z状态

使用nydus zran artifact

使用zran artifact镜像启动容器导致nydusd进程d状态

5.11.1. 现象

  • pod 无法启动，describe po信息
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 33s default-scheduler Successfully assigned default/busybox-nydus-oci-ref to node1 Normal Pulled 31s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 1.0s Normal Pulled 25s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 119.ms Warning Failed 19s (x2 over 25s) kubelet Error: failed to create containerd container: wait until daemon is RUNNING: get daemon state: daemon socket /var/lib/containerd-nydus/socket/cho77etp707gng95ic00/api.sock: not found Normal Pulling 5s (x3 over 32s) kubelet Pulling image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" Normal Pulled 5s kubelet Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 101.ms
  • 查看nydus-snapshotter log
2023-05-26 08:54:19. +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21df07d32a5095a19398", Build Time: "2023-03-09T10:09:03.Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)" [2023-05-26 08:54:19. +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to , maximum  [2023-05-26 08:54:19. +00:00] DEBUG [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/pseudo_fs.rs:161] pseudo fs iterate "/" [2023-05-26 08:54:19. +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_SHA256 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_GZIP | PRESERVED_INLINED_CHUNK_DIGEST [2023-05-26 08:54:19. +00:00] ERROR [rafs/src/metadata/layout/v6.rs:1375] RafsV6Blob: idx 0 invalid ci_digest [2023-05-26 08:54:19. +00:00] ERROR [error/src/error.rs:21] Error: "invalid Rafs v6 blob entry" at rafs/src/metadata/layout/v6.rs:1495 note: enable `RUST_BACKTRACE=1` env to display a backtrace [2023-05-26 08:54:19. +00:00] ERROR [error/src/error.rs:21] Error: Rafs(FillSuperblock(Os { code: 22, kind: InvalidInput, message: "Invalid argument" })) at src/bin/nydusd/daemon.rs:140 note: enable `RUST_BACKTRACE=1` env to display a backtrace [2023-05-26 08:54:19. +00:00] ERROR [src/bin/nydusd/main.rs:664] Failed in starting daemon: Invalid argument (os error 22) Error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" } time="2023-05-26T08:54:19.Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:19.Z" level=error msg="Nydusd cho77etp707gng95ic00 probably not started" time="2023-05-26T08:54:19.Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:19.Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:19.Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:19.Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:20.0Z" level=error msg="Process 14029 has been a zombie" time="2023-05-26T08:54:20.Z" level=error msg="Process 14029 has been a zombie"
  • 查看进程进入z状态
[root@node1 nydus]# ps -ef|grep 14029 root 14029 22492 0 08:54 ? 00:00:00 [nydusd] <defunct> root 16385 15474 0 08:56 pts/2 00:00:00 grep --color=auto 14029

5.11.2. 解决方式

oci ref 要求 nydus-image/nydusd 2.2

5.12. nydus镜像和oci镜像拉取镜像时间相同

nydus镜像容器创建时最开始只会加载最后一层，所以会很快

{ 
    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "digest": "sha256:ff67b2807cdb0005ac0137bb8d5a4cf453feadf56351fa2ecc3acba0", "size": , "annotations": { 
    "containerd.io/snapshot/nydus-bootstrap": "true", "containerd.io/snapshot/nydus-fs-version": "6" } }

对比的时候记得把已经在本地的镜像删除掉

  • 编辑 configmap 增加nydus.toml配置
kubectl edit cm -n nydus-system nydus-snapshotter 
apiVersion: v1 data: config.json: |- ... nydus.toml: |- version = 1 address = "/run/containerd-nydus/containerd-nydus-grpc.sock" daemon_mode = "multiple" cleanup_on_close = false [system] enable = true address = "/run/containerd-nydus/system.sock" [system.debug] daemon_cpu_profile_duration_secs = 5 pprof_address = "" [daemon] nydusd_path = "/usr/local/bin/nydusd" nydusimage_path = "/usr/local/bin/nydus-image" fs_driver = "fusedev" recover_policy = "restart" nydusd_config = "/etc/nydus/config.json" threads_number = 10 [log] log_to_stdout = false level = "debug" log_rotation_compress = true log_rotation_local_time = true log_rotation_max_age = 7 log_rotation_max_backups = 5 log_rotation_max_size = 1 [metrics] address = ":9110" [remote] convert_vpc_registry = false [remote.mirrors_config] dir = "" [remote.auth] enable_kubeconfig_keychain = false kubeconfig_path = "" enable_cri_keychain = false image_service_address = "" [snapshot] enable_nydus_overlayfs = false sync_remove = false [cache_manager] disable = false gc_period = "24h" [image] public_key_file = "" validate_signature = false [experimental] enable_stargz = false enable_referrer_detect = false
  • 重启nydus-snapshotter

去掉 /etc/containerd/config.toml 中的

[plugins."io.containerd.grpc.v1.cri".containerd] discard_unpacked_layers = false disable_snapshot_annotations = false snapshotter = "nydus" [proxy_plugins] [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd

systemctl restart containerd

delete nydus-snapshot容器

再次创建即可

5.12.1. nydus镜像启动 nydus-snapshotter日志

nydus镜像会有.rs 输出，这部分是nydus代码用rust编写

time="2023-05-26T11:04:29.Z" level=info msg="[Mounts] snapshot k8s.io/699/987c0c48cdfcbd8b451e7c892d2bc3c2fa07e2741ae5d003792dff96f9 ID 457 Kind Active" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/700/extract--v57_ sha256:17f262ebfabe0a80a19356d4acdde01515c6488b4baf9f474ba805b205be6bcd\" snapshot id 458" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/701/extract--roQj sha256:4969b12a639d311e01eb2997dca25b3ff4c15297cf1cefc3da416ed90\" snapshot id 459" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/702/extract--ES_o sha256:eeaaea67dfdbce46442b556a6773a7e65ad\" snapshot id 460" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/703/extract--M0p7 sha256:f2a156cd5d860cc6cf21deeac187cef6d8cb5958f08d\" snapshot id 461" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/704/extract--zDfC sha256:977a5025efb33940f21f7bc078a4b449a197bf5a84809d18eafa2ecc\" snapshot id 462" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/705/extract--m4B7 sha256:b9b3afe811ab255d360cb9f6c03db6be9d1a82b10ea1ce\" snapshot id 463" time="2023-05-26T11:04:29.Z" level=info msg="[Commit] snapshot with key \"k8s.io/706/extract--zC2V sha256:dbb29c913fcddebd164aff00387b0cf2ae2605ca7c3ff8ae7fc81e3770c09d6d\" snapshot id 464" time="2023-05-26T11:04:30.00Z" level=info msg="[Commit] snapshot with key \"k8s.io/707/extract--KS4S sha256:1dc2d75a8c5b4572fcaeb4ee74ee4a34f1f557cf380d85560bea88ddf8e656f1\" snapshot id 465" time="2023-05-26T11:04:30.0Z" level=info msg="[Commit] snapshot with key \"k8s.io/708/extract--sAab sha256:2e352e3c2a94ff6a53bda70e8c78f1c0061bc5c09918d42f10ee42ca0ea6e05d\" snapshot id 466" time="2023-05-26T11:04:30.0Z" level=info msg="[Commit] snapshot with key \"k8s.io/709/extract--0fYc sha256:c4655b7eed27aae6af1bbf99061da67f84943ebf11657db9841d2ce34\" snapshot id 467" time="2023-05-26T11:04:30.0Z" level=info msg="[Commit] snapshot with key \"k8s.io/710/extract--IVNU sha256:c1775d8affc2abbc45ed4051e72fa8083f624f1a52f938e3dd4017ff2a\" snapshot id 468" time="2023-05-26T11:04:30.0Z" level=info msg="[Commit] snapshot with key \"k8s.io/711/extract--8xBS sha256:7cd39c27735b21390c4f959ca1e12d38c2b73a5f173f5ffd1ee313fee85e317e\" snapshot id 469" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/712/extract--LQ31 sha256:ecf3a25d66b8a4f683b381ec2314cd293a61daedce7f1d8f\" snapshot id 470" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/713/extract--x2We sha256:a369a3f699eb345cb9655d22d8850e1713e20766dcc5fc6cf2d\" snapshot id 471" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/714/extract--w72W sha256:856e93de3351e85ffc11f1c6fcaf4d7de56c50a4a40936ea6dc\" snapshot id 472" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/715/extract--OaK7 sha256:bfdfb86a11099f8dfdcecb08a46dee87408cb00dedbb69d8c41bd4524e661fb6\" snapshot id 473" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/716/extract--MjNO sha256:205b438eb857a4428cdfc1a492c3e9e445a7f6aad780f69b082ec9a2\" snapshot id 474" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/717/extract--U7eo sha256:a72248b4d79a6915f529bb00b97b576fecd81c0ebb125a02dd76f91169\" snapshot id 475" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/718/extract--32MC sha256:f0fc4ec3e74f3762cc5c68435c8c93fb92fcac6ae34053cae00ae90dcafceab6\" snapshot id 476" time="2023-05-26T11:04:30.Z" level=info msg="[Commit] snapshot with key \"k8s.io/719/extract--VBbE sha256:3be1111cf1b2f35366acf9daf4860fe8415c7be30faae3de1fde0f3\" snapshot id 477" time="2023-05-26T11:04:30.Z" level=info msg="Prepares active snapshot k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481, nydusd should start afterwards" key=k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481 parent="k8s.io/720/sha256:3be1111cf1b2f35366acf9daf4860fe8415c7be30faae3de1fde0f3" time="2023-05-26T11:04:30.Z" level=info msg="nydusd command: /usr/local/bin/nydusd fuse --config /var/lib/containerd-nydus/config/cho94flp707i10bb5ddg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/477/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/477/mnt --apisock /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock --log-level debug" [2023-05-26 11:04:30. +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21df07d32a5095a19398", Build Time: "2023-03-09T10:09:03.Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)" [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to , maximum  [2023-05-26 11:04:30. +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_ZSTD | PRESERVED_INLINED_CHUNK_DIGEST [2023-05-26 11:04:30. +00:00] INFO [storage/src/backend/connection.rs:263] backend config: ConnectionConfig { proxy: ProxyConfig { url: "", ping_url: "", fallback: false, check_interval: 5, use_http: false }, mirrors: [], skip_verify: false, timeout: 5, connect_timeout: 5, retry_limit: 2 } [2023-05-26 11:04:30. +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time:  [2023-05-26 11:04:30. +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time:  [2023-05-26 11:04:30. +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time:  [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/fs_service.rs:206] Rafs imported [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/fs_service.rs:102] Rafs mounted at / [2023-05-26 11:04:30. +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/transport/fusedev/linux_session.rs:337] mount source rafs dest /var/lib/containerd-nydus/snapshots/477/mnt with fstype fuse opts default_permissions,allow_other,fd=3,rootmode=40000,user_id=0,group_id=0 fd 3 [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Init to Ready, input [Mount], output [None] [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Ready to Running, input [Start], output [Some(StartService)] [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/fusedev.rs:318] start 4 fuse servers [2023-05-26 11:04:30. +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:614] FUSE INIT major 7 minor 23 [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/main.rs:660] Fuse daemon started! [2023-05-26 11:04:30. +00:00] INFO [src/bin/nydusd/api_server_glue.rs:406] HTTP API server running at /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock [2023-05-26 11:04:30. +00:00] INFO [api/src/http.rs:911] http server started [2023-05-26 11:04:30. +00:00] INFO [api/src/http.rs:816] <--- Get Uri { string: "/api/v1/daemon" } [2023-05-26 11:04:30. +00:00] INFO [api/src/http.rs:821] ---> Get Status Code: OK, Elapse: Ok(146.605µs), Body Size: 866 time="2023-05-26T11:04:30.Z" level=info msg="Nydus remote snapshot 477 is ready" time="2023-05-26T11:04:30.Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]" time="2023-05-26T11:04:30.Z" level=info msg="Subscribe daemon cho94flp707i10bb5ddg liveness event, path=/var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock." time="2023-05-26T11:04:30.Z" level=info msg="[Mounts] snapshot k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481 ID 478 Kind Active" time="2023-05-26T11:04:30.Z" level=info msg="Nydus remote snapshot 477 is ready" time="2023-05-26T11:04:30.Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]" time="2023-05-26T11:04:31.Z" level=info msg="[Mounts] snapshot k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481 ID 478 Kind Active" time="2023-05-26T11:04:31.Z" level=info msg="Nydus remote snapshot 477 is ready" time="2023-05-26T11:04:31.Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]" time="2023-05-26T11:04:31.Z" level=info msg="[Mounts] snapshot k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481 ID 478 Kind Active" time="2023-05-26T11:04:31.Z" level=info msg="Nydus remote snapshot 477 is ready" time="2023-05-26T11:04:31.Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]" time="2023-05-26T11:04:31.Z" level=info msg="[Mounts] snapshot k8s.io/721/addfa90857bb598a256abde9e57c23b0a214a5481 ID 478 Kind Active" time="2023-05-26T11:04:31.Z" level=info msg="Nydus remote snapshot 477 is ready" time="2023-05-26T11:04:31.Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

5.12.2. oci镜像启动 nydus-snapshotter日志

time="2023-05-26T11:06:30.Z" level=info msg="[Mounts] snapshot k8s.io/722/3d4aa771e42ef99f0d65ceceaf08bc260eecf2ec3461f7 ID 479 Kind Active" time="2023-05-26T11:06:33.Z" level=info msg="[Commit] snapshot with key \"k8s.io/723/extract--VjiP sha256:8cbe4b54fa88d8fc0198ea0cc3a5432aea41573e6a0ee26eca8c79f9fbfa40e3\" snapshot id 480" time="2023-05-26T11:06:34.Z" level=info msg="[Commit] snapshot with key \"k8s.io/725/extract--Zc-C sha256:61aba3d78f453ab50b4c0bebb47b6d6dc290f57752d88b737f75a9835c\" snapshot id 481" time="2023-05-26T11:06:37.Z" level=info msg="[Commit] snapshot with key \"k8s.io/727/extract--GvJe sha256:d7027bd638e20f68a20d6ae247f2dda259aed1b46a6737bb70de28fa781a1769\" snapshot id 482" time="2023-05-26T11:06:38.00Z" level=info msg="[Commit] snapshot with key \"k8s.io/729/extract--jE_Q sha256:f116aef2f51af904cebd6baaf86af8540f0837f5081dff235b29135d32\" snapshot id 483" time="2023-05-26T11:06:38.Z" level=info msg="[Commit] snapshot with key \"k8s.io/731/extract--aGRw sha256:ffc75289c3fd7cc9423f8b42f1b9cf3d5f1ed3703f0ec83e0\" snapshot id 484" time="2023-05-26T11:06:38.Z" level=info msg="[Commit] snapshot with key \"k8s.io/733/extract--FKvu sha256:41ca5e8fdea7603d4b22d1bd5dacf18cd8ae4a337cfb38329be6428\" snapshot id 485" time="2023-05-26T11:06:38.Z" level=info msg="[Commit] snapshot with key \"k8s.io/735/extract--DHtG sha256:748bab432b39955fe0fb9e2c838c1d82613f6331d4dd06806c6eb08a7\" snapshot id 486" time="2023-05-26T11:06:38.Z" level=info msg="[Commit] snapshot with key \"k8s.io/737/extract--kLLd sha256:affc46ccf46108f1a71012bda8de401ab47acdde\" snapshot id 487" time="2023-05-26T11:06:38.Z" level=info msg="[Commit] snapshot with key \"k8s.io/739/extract--OX_U sha256:290e04ebe112a50211cc52fcdba1e90c13c86b7d28fc9d5810fb4ff66\" snapshot id 488" time="2023-05-26T11:06:39.Z" level=info msg="[Commit] snapshot with key \"k8s.io/741/extract--06mH sha256:f56f832babd23ff7bd23bfff52c30a25fff73521d22231ae96d2\" snapshot id 489" time="2023-05-26T11:06:39.Z" level=info msg="[Commit] snapshot with key \"k8s.io/743/extract--ZzGp sha256:7edd0ab32d42a30e71ac050a4b0b52e52637a109f125e07a\" snapshot id 490" time="2023-05-26T11:06:39.Z" level=info msg="[Commit] snapshot with key \"k8s.io/745/extract--20YV sha256:8e0edcf4b4ba88591fb18a3189e4ef180a3e907b3d3d374cecc59210a\" snapshot id 491" time="2023-05-26T11:06:39.Z" level=info msg="[Commit] snapshot with key \"k8s.io/747/extract--wwaO sha256:0c386a4066a2badc0f4fdb017f83fc5f5c8ebc057\" snapshot id 492" time="2023-05-26T11:06:40.0Z" level=info msg="[Commit] snapshot with key \"k8s.io/749/extract--gUBX sha256:1f3cc06dcc0baaac488df5f027\" snapshot id 493" time="2023-05-26T11:06:40.Z" level=info msg="[Commit] snapshot with key \"k8s.io/751/extract--xwQT sha256:e723aac1be491e74d1de67af90ddff842b1428f56cc2b36e036c9bb1bf76fd99\" snapshot id 494" time="2023-05-26T11:06:40.Z" level=info msg="[Commit] snapshot with key \"k8s.io/753/extract--GuvE sha256:7ac16f8bd1c8be5d3085c99fa945d1545f8b6e4969cc9ba7f932e\" snapshot id 495" time="2023-05-26T11:06:40.Z" level=info msg="[Commit] snapshot with key \"k8s.io/755/extract--p-5u sha256:4c8bfefa8feaab6cfbb30bdb73dba02d105a432d4c3cdd3b4c02ba53d1\" snapshot id 496" time="2023-05-26T11:06:40.Z" level=info msg="[Commit] snapshot with key \"k8s.io/757/extract--v1Ai sha256:239e5e3542f395b783fe8f614b0c2e5fae5acff8d297f81b96bfb65384\" snapshot id 497" time="2023-05-26T11:06:41.Z" level=info msg="[Commit] snapshot with key \"k8s.io/759/extract--6llW sha256:bb99dc5dfa254bc0ee14ce436bbb0b9010ad90f75dd4e9034\" snapshot id 498" time="2023-05-26T11:06:41.Z" level=info msg="[Commit] snapshot with key \"k8s.io/761/extract--OV2m sha256:057bf4d9b4e6fdb652c002e7fd7e34dea9a6955fd02b\" snapshot id 499" time="2023-05-26T11:06:41.Z" level=info msg="[Commit] snapshot with key \"k8s.io/763/extract--B0nU sha256:92125a743f1af17a7e6e4dcaaa76483e6ff3f1b70abb8c3833ed\" snapshot id 500" time="2023-05-26T11:06:41.Z" level=info msg="[Mounts] snapshot k8s.io/765/20ef1230a38b64c5677fd8eb75a06a0dcb364f78fca4e8f10 ID 501 Kind Active" time="2023-05-26T11:06:45.Z" level=info msg="[Mounts] snapshot k8s.io/765/20ef1230a38b64c5677fd8eb75a06a0dcb364f78fca4e8f10 ID 501 Kind Active" time="2023-05-26T11:06:45.Z" level=info msg="[Mounts] snapshot k8s.io/765/20ef1230a38b64c5677fd8eb75a06a0dcb364f78fca4e8f10 ID 501 Kind Active" time="2023-05-26T11:06:45.Z" level=info msg="[Mounts] snapshot k8s.io/765/20ef1230a38b64c5677fd8eb75a06a0dcb364f78fca4e8f10 ID 501 Kind Active"

6. 参考

  • 参考链接
    • docker使用nydus启动容器暂不支持
    • buildkit-laily-fetch
    • How to understand the ‘task’ concept in containerd
    • Nerdctl 原生支持 Nydus 加速镜像
    • Nydus | 容器镜像基础
    • Install nydus for containerd
    • how to check image manifest in containerd?
    • run_nydus_in_kubernetes
    • nydus社区钉钉支持
    • v2list.manifest
  • 主要涉及的github项目
    • image-service
    • moby/buildkit
    • nydusaccelerator/buildkit
    • nydus-snapshotter
  1. nydus-snapshotter ↩︎
  2. buildkitd.toml ↩︎
  3. nydus-design ↩︎
  4. nydus ↩︎
  5. run_nydus_in_kubernetes ↩︎
  6. buildkit-k8s-example ↩︎
  7. How-to-understand-the-‘task’-concept-in-containerd ↩︎
  8. how-to-check-image-manifest-in-containerd? ↩︎
  9. opencontainers/image-spec ↩︎

免责声明：本站所有文章内容,图片，视频等均是来源于用户投稿和互联网及文摘转载整编而成，不代表本站观点，不承担相关法律责任。其著作权各归其原作者或其出版社所有。如发现本站有涉嫌抄袭侵权/违法违规的内容,侵犯到您的权益，请在线联系站长,一经查实,本站将立刻删除。 本文来自网络,若有侵权，请联系删除，如若转载，请注明出处：https://haidsoft.com/138972.html

赞 (0)
老牧童老牧童
0
上一篇 2025-06-09 18:26
下一篇 2025-06-09 18:33

相关推荐

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注

关注微信