制品的上传拉取
Harbor 是一个开源的企业级制品(Artifact)管理仓库,用于存储和管理 Docker 镜像、Helm Charts、OCI 容器镜像、ChartMuseum 等制品。
1. docker或nerdctl作为客户端推送拉取容器镜像
以下记录docker推送拉取harbor仓库镜像的一般步骤,及一些可能用到的配置。nerdctl客户端基本兼容docker,操作过程类似,只需执行命令时将docker改为nerdctl即可。
操作步骤如下:
- 确认已经安装docker或者nerdctl
本文使用的Docker版本如下:
[root@k8s-node1 ~]# docker version
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:32:04 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.4
API version: 1.45 (minimum version 1.24)
Go version: go1.21.11
Git commit: de5c9cf
Built: Wed Jun 5 11:31:02 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.33
GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
- 配置证书或非安全仓库信任
如果Harbor是https部署,需要给docker配置根证书:
# 生成Harbor根证书,k8s环境部署的Harbor执行如下如下命令生成。也可以在界面下载:Harbor管理界面-系统管理-配置管理-系统设置-镜像库根证书。
[root@k8s-master1 ~]# kubectl -n harbor get secrets harbor-ingress -o jsonpath="{.data.ca\.crt}" | base64 -d >ca.crt
# 创建证书存放目录,主要文件夹同Harbor的external url命名,如果是ip地址,同理建立192.168.1.1类似的文件夹
[root@k8s-master1 ~]# mkdir -p /etc/docker/certs.d/harbor.test.com
[root@k8s-master1 ~]# ls /etc/docker/certs.d/harbor.test.com/ca.crt
/etc/containerd/certs.d/harbor.test.com/ca.crt
如果Harbor是http部署,在 /etc/docker/daemon.json
文件中加入如下配置:
{
"insecure-registries": [
"harbor.test.com"
]
}
修改之后重启docker生效:
systemctl restart docker
说明:
知识配置证书部分不需要重启docker
- 登录镜像仓库
[root@k8s-master1 ~]# docker login harbor.test.com -uadmin -p Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
- 拉取镜像
执行以下命令可将仓库中的镜像拉取到本地。
docker pull Harbor_address/ProjectName/Repsitory[:Tag]
其中:
Harbor_address
表示 Harbor 镜像仓库地址。ProjectName
表示项目名称。Repsitory
表示镜像仓库名称。Tag
表示镜像标签,通常表示镜像版本。
一般如果使用harbor
作为镜像仓库,界面都会提供一键复制拉取镜像的命令。
此外除了Tag用于表示镜像版本之外,还可以用摘要(digest)进行镜像区分,以下两种方式的简单对比:
特性 | 标签(Tag) | 摘要(Digest) |
---|---|---|
用途 | 人类可读的标识符 | 镜像内容的唯一标识符 |
格式 | 镜像名称:标签 (如: nginx:latest ) | 镜像名称@sha256:摘要值 (如: nginx@sha256:abcdef... ) |
可变性 | 可变:可以重新分配给不同的镜像 | 不可变:唯一对应特定的镜像内容 |
易读性 | 高:易于识别和管理不同版本 | 低:哈希值较长且难以阅读 |
用途 | 开发和测试环境,可以使用latest 标签来拉取最新的镜像。 | 生产环境和严格控制镜像内容的场景,确保部署的镜像内容一致且不变。例如,CI/CD流水线中使用摘要来确保部署的镜像内容与测试时一致。 |
示例 | nginx:latest , nginx:1.21.3 | nginx@sha256:abcdef123456... |
拉取命令一般可以到Harbor的项目-镜像仓库中,针对具体的镜像可以复制拉取命令:
docker pull harbor.test.com/server/nginx@sha256:4a50ed86d8c86e35f530d4a168173677a192177eed14146fbb5728b1b3a2d4de
- 推送镜像
执行以下命令在项目中标记镜像:
docker tag SourceImage[:Tag] Harbor_address/ProjectName/Repsitory[:Tag]
其中:SourceImage
表示本地的镜像。
执行以下命令将镜像上传到 Harbor 镜像仓库:
docker push Harbor_address/ProjectName/Repsitory[:Tag]
docker login登录Harbor后,执行推送:
[root@k8s-master1 ~]# docker push harbor.test.com/flannel/flannel-cni-plugin:v1.4.1-flannel1
The push refers to repository [harbor.test.com/flannel/flannel-cni-plugin]
756a157559aa: Layer already exists
8980900bae6f: Layer already exists
v1.4.1-flannel1: digest: sha256:ece4198ce10c4d4ba1a302fc4d66dc073598edae2b3e8e7ca8957fadf1f28b63 size: 738
2. ctr作为客户端推送拉取容器镜像
ctr是containerd运行时自带的客户端,部分环境中如果没有安装docker或者nerdctl客户端,可以选用ctr作为客户端进行镜像的上传下载。
本文使用的containerd版本如下:
[root@e2etest ~]# ctr -v
ctr containerd.io 1.6.33
[root@e2etest ~]# containerd -v
containerd containerd.io 1.6.33 d2d58213f83a351ca8f528a95fbd145f5654e957
2.1 配置证书
# 生成Harbor根证书,k8s环境部署的Harbor执行如下如下命令生成。也可以在界面下载:Harbor管理界面-系统管理-配置管理-系统设置-镜像库根证书。
[root@k8s-master1 ~]# kubectl -n harbor get secrets harbor-ingress -o jsonpath="{.data.ca\.crt}" | base64 -d >ca.crt
# 创建证书存放目录,主要文件夹同Harbor的external url命名,如果是ip地址,同理建立192.168.1.1类似的文件夹
[root@k8s-master1 ~]# mkdir -p /etc/containerd/certs.d/harbor.test.com/
[root@k8s-master1 ~]# ls /etc/containerd/certs.d/harbor.test.com/ca.crt
/etc/containerd/certs.d/harbor.test.com/ca.crt
备注
https部署的Harbor操作此步骤。根证书也可以从Harbor界面–系统管理–配置管理–系统设置–镜像库根证书位置进行下载。
2.2 修改配置文件
修改containerd配置文件,如果没有通过containerd config default > /etc/containerd/config.toml
生成。
备注:
如果是docker单机环境,最新的docker会同时安装containerd,默认的配置文件config.toml内容较少,可以使用上面命令生成新的配置文件。
vim /etc/containerd/config.toml
在 /etc/containerd/config.toml
文件中对应位置加入如下的配置:
...
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.test.com".tls] # harbor.test.com为我的仓库地址
insecure_skip_verify = true # 上述的证书为自签名证书,这里设置跳过证书不安全验证
ca_file = "/etc/containerd/certs.d/harbor.test.com/ca.crt" # 这一步应该和上一步作用类似,没有进一步验证。
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.test.com".auth] # harbor的认证信息
username = "admin"
password = "Harbor12345"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.test.com"]
endpoint = ["https://harbor.test.com"]
...
配置之后需要重启 containerd
:
systemctl restart containerd
containerd 2.0版本之前和之后的配置方式不同,本文使用的是2.0之前的版本。关于该部分的内容可以参考:https://github.com/containerd/containerd/blob/main/docs/cri/registry.md
2.3 ctr拉取推送镜像
正常来讲按照上述配置后,ctr可以正常推送拉取Harbor仓库的镜像。本文配置后,拉取推送还是报证书签名机构未知,最后加了--skip-verify
才执行成功。
[root@e2etest ~]# ctr i pull harbor.test.com/perf/test-del:v1
WARN[0000] DEPRECATION: The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
INFO[0000] trying next host error="failed to do request: Head \"https://harbor.test.com/v2/perf/test-del/manifests/v1\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=harbor.test.com
ctr: failed to resolve reference "harbor.test.com/perf/test-del:v1": failed to do request: Head "https://harbor.test.com/v2/perf/test-del/manifests/v1": tls: failed to verify certificate: x509: certificate signed by unknown authority
[root@e2etest ~]# ctr i pull harbor.test.com/perf/test-del:v1 --skip-verify
WARN[0000] DEPRECATION: The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
harbor.test.com/perf/test-del:v1: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:a7fad8c1072a21345b8757f34ac75f0d9aacb06f6daa41688c7a267f44fea24a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d63decd1625fd2c540d723a81320df7d9f570f445c1a3ef976ca2675d154648e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:47f50659768025a3e54da267f00011a8fa28a9e6a61c170f34b2e866367e3679: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:184984d263c287122b19c49897027f6fa746f41abb0f1cec0611ed919e4041d5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2a715ab8d96eb980e74a015f299d9c14156ebfb94230c7355090d84a84043a29: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8663a019712297de33c855c6322f652dda7d63ebc9c52d7025b64ef252277372: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4e6c916f777cb68db9f51fb0b27783d0d945c4f2232fac1f0c5712267e53a0ee: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:51031a802bd2cceb2441df12a78d34e990bf345b361d83803da95869bdc3da16: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.2 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:a7fad8c1072a21345b8757f34ac75f0d9aacb06f6daa41688c7a267f44fea24a...
done: 9.012416ms
[root@e2etest ~]# ctr i push harbor.test.com/perf/test-del:v1 --skip-verify
WARN[0000] DEPRECATION: The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use `config_path` instead.
manifest-sha256:a7fad8c1072a21345b8757f34ac75f0d9aacb06f6daa41688c7a267f44fea24a: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:184984d263c287122b19c49897027f6fa746f41abb0f1cec0611ed919e4041d5: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.4 s total: 5.6 Ki (14.0 KiB/s)
备注:
如果Harbor是http协议访问,配置文件中无需配置tls相关设置,ctr执行推送拉取时加flag
--plain-http
。
3. podman作为客户端推送拉取容器镜像
podman是更新的容器技术,大部分场景下都兼容docker,当docker做了上述配置后,podman即可直接从Harbor拉取或推送镜像。
[root@e2etest ~]# podman login harbor.test.com
Authenticating with existing credentials...
Existing credentials are valid. Already logged in to harbor.test.com
[root@e2etest ~]# podman pull harbor.test.com/perf/test-del:v1
[root@e2etest ~]# podman push harbor.test.com/perf/test-del:v1
当然podman也有自己的仓库配置文件/etc/containers/registries.conf
,如果是环境中只有podman,则需要单独配置。这里就不细展开了,可以查看podman相关文章。
4. Helm Charts的拉取推送
Charts的推送依赖于helm,本文使用的helm版本如下:
[root@e2etest ~]# helm version
version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.3"}
4.1 上传 Helm Chart 到 Harbor
使用 helm push
插件将 Helm Chart 上传到 Harbor 中的 ChartMuseum:
# 登录helm仓库,如果没有配置tls证书加--insecure
root@master1:~/harbor# helm registry login harbor.test.com --insecure
Username: admin
Password:
Login Succeeded
# 提前在harbor中创建harbor项目。上传不再支持UI界面,必须使用helm push。需要提前在harbor中创建名为harbor的project
root@master1:~/harbor# helm push harbor-1.15.0.tgz oci://harbor.test.com/harbor
Error: failed to do request: Head "https://harbor.test.com/v2/harbor/harbor/blobs/sha256:c51d391fc1d26224da59d97cc0428ecc9be64c9ca5ba9381f4e48f696bf41dc5": tls: failed to verify certificate: x509: certificate signed by unknown authority
# 添加--insecure-skip-tls-verify跳过tls证书验证。
root@master1:~/harbor# helm push harbor-1.15.0.tgz oci://harbor.test.com/harbor --insecure-skip-tls-verify
Pushed: harbor.test.com/harbor/harbor:1.15.0
Digest: sha256:2bb53e13ae59d5478ca90734b2c267096a1919a9a63875e583b87efff76a4532
4.2 从 Harbor 拉取 Helm Chart
使用 helm fetch
命令从 Harbor 中的 ChartMuseum 拉取 Helm Chart:
helm fetch harbor.example.com/chartrepo/mychart
# 下载命令web界面可以复制
helm pull oci://harbor.test.com/harbor/harbor --version 1.15.0
4.3 问题记录
- 如果Harbor为http部署,helm无法登录Harbor仓库
root@e2etest harbor]# helm registry login -u admin 10.210.10.210
Password:
INFO[0003] Error logging in to endpoint, trying next endpoint error="Get \"https://10.210.10.210/v2/\": dial tcp 10.210.10.210:443: connect: connection refused"
Error: Get "https://10.210.10.210/v2/": dial tcp 10.210.10.210:443: connect: connection refused
问题遗留,后续跟踪。