containerd 使用harbor

一. harbor安装配置

1.1 下载harbor

wget https://github.com/goharbor/harbor/releases/download/v2.11.0/harbor-offline-installer-v2.11.0.tgz

https://objects.githubusercontent.com/github-production-release-asset-2e65be/50613991/ff4b1471-df37-4174-a332-a7e698092ca8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240607%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240607T072025Z&X-Amz-Expires=300&X-Amz-Signature=a74a04769f1ba7f390199b96434d2429be7bc55340d60c1e5659ac2b54d41f1f&X-Amz-SignedHeaders=host&actor_id=12129691&key_id=0&repo_id=50613991&response-content-disposition=attachment%3B%20filename%3Dharbor-offline-installer-v2.11.0.tgz&response-content-type=application%2Foctet-stream

下载docker-compose

curl -L "https://mirror.ghproxy.com/https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod  +x /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

安装 docker
略

1.2 配置harbor
解压

tar -zxvf harbor-offline-installer-v2.11.0.tgz

cd harbor
编辑 harbor.yml
设置 hostname: harbor.cctbb.com

1.3 配置harbor https
生成证书颁发机构证书及私钥

 openssl genrsa -out ca.key 4096
 openssl req -x509 -new -nodes -sha512 -days 3650 \
  -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.cctbb.com" \
  -key ca.key \
  -out ca.crt

生成服务器私钥及证书签名请求(CSR)
``
openssl genrsa -out harbor.cctbb.com.key 4096

生成证书签名请求

openssl req -sha512 -new
-subj “/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.cctbb.com”
-key harbor.cctbb.com.key
-out harbor.cctbb.com.csr

生成 x509 v3 扩展文件。
无论使用 FQDN 还是 IP 地址连接到Harbor服务器，都必须创建此文件，以便keyid为的 Harbor 主机生成符合主题备用名称 (SAN) 和 x509 v3 的证书扩展要求。

cat v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.cctbb.com
DNS.2=cctbb.com
DNS.3=harbor

使用该v3.ext文件为 Harbor 服务器生成证书。

openssl x509 -req -sha512 -days 3650
-extfile v3.ext
-CA ca.crt -CAkey ca.key -CAcreateserial
-in harbor.cctbb.com.csr
-out harbor.cctbb.com.crt
Signature ok
subject=C = CN, ST = Shanghai, L = Shanghai, O = SmartX, OU = Lab, CN = harbor.cctbb.com
Getting CA Private Key

将 harbor.cctbb.com.crt 转换为 harbor.cctbb.com.cert , 供 Docker 使用。Docker 守护进程将.crt文件解释为 CA 证书，.cert将文件解释为客户端证书。

openssl x509 -inform PEM -in harbor.cctbb.com.crt -out harbor.cctbb.com.cert

配置 harbor yml文件，注意需要修改hostname与指定证书位置。

vim harbor.yml
…………

https related config

https:

https port for harbor, default is 443

port: 443

The path of cert and key files for nginx

certificate: /apps/harbor/cert/harbor.cctbb.com.cert
private_key: /apps/harbor/cert/harbor.cctbb.com.key

enable strong ssl ciphers (default: false)

strong_ssl_ciphers: false

执行安装

sh install.sh

登录
http://192.168.2.200
admin/Harbor12345

配置域名解析  
使用域名访问
http://harbor.cctbb.com


1.4 安装nerdctl
crictl 工具在镜像操作功能有缺陷,使用nerdctl

https://github.com/containerd/nerdctl/releases/tag/v1.7.6

tar xf nerdctl-1.7.6-linux-amd64.tar.gz -C /usr/bin

下载镜像

nerdctl pull nginx:alpine
nerdctl pull registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx

[root@ops-harbor ~]# nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx latest 423210a5903e 36 seconds ago linux/amd64 111.3 MiB 42.0 MiB

为该镜像打TAG

nerdctl tag registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx:latest harbor.cctbb.com/library/nginx:latest

[root@ops-harbor ~]# nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
harbor.cctbb.com/library/nginx latest 423210a5903e 4 seconds ago linux/amd64 111.3 MiB 42.0 MiB
registry.cn-hangzhou.aliyuncs.com/acs-sample/nginx latest 423210a5903e 3 minutes ago linux/amd64 111.3 MiB 42.0 MiB

 登录harbor

nerdctl login harbor.cctbb.com
Enter Username: admin
Enter Password:

如果报错：

[root@ops-harbor harbor]# nerdctl login harbor.cctbb.com
Enter Username: admin
Enter Password:
ERRO[0007] failed to call tryLoginWithRegHost error=“failed to call rh.Client.Do: Get “https://harbor.cctbb.com/v2/”: tls: failed to verify certificate: x509: certificate signed by unknown authority” i=0
FATA[0007] failed to call rh.Client.Do: Get “https://harbor.cctbb.com/v2/”: tls: failed to verify certificate: x509: certificate signed by unknown authority

遇到的错误信息表明服务器 harbor.cctbb.com 的证书是由未知的证书颁发机构（CA）签署的，这意味着默认情况下，您的系统无法验证该证书。
以下是解决此问题的步骤：
安装证书颁发机构（CA）证书：如果Harbor服务器使用的是自签名证书或由私人CA签署的证书，您需要将CA证书添加到系统的受信任证书存储中。
获取CA证书：从您的Harbor服务器管理员那里获取CA证书（通常是 .crt 文件）。
将CA证书添加到系统信任存储：

对于RHEL/CentOS：
sudo cp /path/to/ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

对于Ubuntu/Debian：
sudo cp /path/to/ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

之后再登录

[root@ops-harbor cert]# nerdctl login harbor.cctbb.com
Enter Username: admin
Enter Password:
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@ops-harbor cert]#

上传镜像
上传到 harbor 仓库
```
nerdctl push harbor.cctbb.com/library/nginx:latest

至此 harbor 配置完成

二. 配置containerd 使用harbor
2.1 生成配置文件

mkdir -p /etc/containerd/ >containerd config default
 /etc/containerd/config.toml

此配置文件在安装containerd时已经配置

2.2 修改配置
大概在162行开始

vim +162 /etc/containerd/config.toml 
161 [plugins."io.containerd.grpc.v1.cri".registry]
162 config_path = "/etc/containerd/certs.d" #修改该行的配置信息 ...

创建该目录
上面的目录+harbor仓库地址

mkdir -p /etc/containerd/certs.d/harbor.cctbb.com

编辑文件hosts.toml

vim /etc/containerd/certs.d/harbor.cctbb.com/hosts.toml

server = "http://harbor.cctbb.com"
[host."http://harbor.cctbb.com"]
  capabilities = ["pull", "resolve", "push"]
skip_verify = true

重启服务

systemctl restart containerd.service

2.3 验证

[root@k8s-node01 containerd]# crictl pull harbor.cctbb.com/library/nginx:latest
Image is up to date for sha256:e4e6d42c70b3f79c5d57c170526592168992eb3303a6594c439302fabd92d9a3
[root@k8s-node01 containerd]# crictl images
IMAGE                                                TAG                 IMAGE ID            SIZE
docker.io/calico/cni                                 v3.27.3             6527a35581401       196MB
docker.io/calico/node                                v3.27.3             5c6ffd2b2a1d0       340MB
harbor.cctbb.com/library/nginx                       latest              e4e6d42c70b3f       44.1MB
registry.aliyuncs.com/google_containers/coredns      v1.11.1             cbb01a7bd410d       18.2MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.30.1             747097150317f       29MB
registry.aliyuncs.com/google_containers/pause        3.9                 e6f1816883972       322kB