云计算k8s集群部署

一、前期准备

准备4台虚拟主机harbor、k8s-m、k8s-n1、k8s-n2

分别配置ip

harbor        172.25.254.254

k8s-m         172.25.254.100

k8s-n1        172.25.254.10

k8s-n2        172.25.254.20

将压缩包 harbor-offline-installer-v2.5.4.tgz放入harbor /root目录

二.构建harbor镜像仓库

1.部署docker本地仓库

在harbor主机上

配置docker镜像源

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
> [docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0
EOF

安装阿帕奇并修改监听端口

[root@harbor ~]# dnf install httpd createrepo -y

[root@harbor ~]# mkdir  /var/www/html/docker/ -p
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf

[root@harbor ~]# systemctl enable --now httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.

安装docker-ce

[root@harbor ~]# dnf  install docker-ce --downloadonly  --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/
[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
> [docker]
name = docker
baseurl = http://172.25.254.254:4444/docker
gpgcheck = 0
EOF
[root@harbor ~]# dnf install docker-ce -y

2.在harbor仓库节点安装docker

[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf

加载配置文件

[root@harbor ~]# systemctl restart  systemd-modules-load.service
[root@harbor ~]# sysctl  --system

为docker.service配置开机启动

[root@harbor ~]# vim /lib/systemd/system/docker.service

[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.

3.生成key

[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey  rsa:4096 \
-nodes -sha256 -keyout /data/certs/timinglee.org.key \
-addext "subjectAltName = DNS:reg.timinglee.org" \
-x509 -days 365 -out /data/certs/timinglee.org.crt


You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Shanxi
Locality Name (eg, city) [Default City]:Xi'an
Organization Name (eg, company) [Default Company Ltd]:kubernetes
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.timinglee.org
Email Address []:xxx

4.编辑harbor配置文件

解压harbor压缩包并修改配置文件(更改hostname、修改key的目录、更改管理员密码)

[root@harbor ~]# tar zxf  harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

随后启动安装脚本完成安装

./install.sh --with-chartmuseum

编写启动脚本

[root@harbor harbor]# vim /lib/systemd/system/harbor.service

[root@harbor harbor]# docker compose  down

’

设置开机启动

[root@harbor harbor]# systemctl enable --now harbor
Created symlink /etc/systemd/system/multi-user.target.wants/harbor.service → /usr/lib/systemd/system/harbor.service.

5.启动并验证

[root@harbor harbor]# mkdir  /etc/docker/certs.d/reg.timinglee.org/ -p
[root@harbor harbor]# cp /data/certs/timinglee.org.crt  /etc/docker/certs.d/reg.timinglee.org/ca.crt
[root@harbor harbor]# vim /etc/hosts

[root@harbor harbor]# vim /etc/hosts
[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker compose up -d

关闭防火墙

[root@harbor harbor]# systemctl disable --now firewalld.service
Removed "/etc/systemd/system/multi-user.target.wants/firewalld.service".
Removed "/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service".

测试登录

[root@harbor harbor]# docker login  reg.timinglee.org -u admin
Password:

三、kubernetes的部署【利用Ansible部署kubernetes运行环境】

先在所有主机上建立解析(可以先修改harbor主机上的hosts后发给其他主机)

[root@harbor harbor]# vim /etc/hosts

[root@harbor harbor]# scp /etc/hosts root@172.25.254.100:/etc/
[root@harbor harbor]# scp /etc/hosts root@172.25.254.10:/etc/
[root@harbor harbor]# scp /etc/hosts root@172.25.254.20:/etc/

所有主机关闭swap

systemctl disable --now  swap.target
systemctl mask swap.target
sed '/swap/s/^/#/g' -i /etc/fstab

1.安装ansible

先配置epel

[root@harbor ~]# vim /etc/yum.repos.d/epel.repo
[epel]
name = epel
baseurl = https://mirrors.aliyun.com/epel/9/Everything/x86_64/
gpgcheck = 0

再安装ansible

[root@harbor harbor]# dnf install ansible-core.x86_64 ansible.noarch -y

2.配置所需用户

[root@harbor ~]# useradd  devops
[root@harbor ~]# echo lee | passwd --stdin devops

切换到该用户并创建ansible目录

[root@harbor harbor]# su - devops
[devops@harbor ~]$ mkdir  ansible
[devops@harbor ~]$ cd ansible/

3.生成ansible的临时配置文件并建立运行环境

[devops@harbor ansible]$ vim inventory

[devops@harbor ansible]$ vim ansible.cfg

运行测试

远程建立用户

[devops@harbor ansible]$ ansible all -m user -a "name=devops state=present"
[devops@harbor ansible]$ ansible all -m shell -a 'echo "lee" | passwd --stdin devops'
[devops@harbor ansible]$ ansible all -m lineinfile -a 'path=/etc/sudoers line="devops    ALL=(ALL)    NOPASSWD: ALL" insertafter=EOF'

4.生成ansible的主配置文件并测试

[devops@harbor ansible]$ vim ansible.cfg

测试能否切换为root用户

5.利用ansible剧本部署docker

[devops@harbor ansible]$ vim kubernetes-devops.yml
-----------------------------------------------------------------------------------------
- name: devops k8s from ansible
  hosts: all
  tasks:
  - name: setup docker repo
    yum_repository:
      name: docker
      description: docker
      baseurl: http://172.25.254.254:4444/docker
      file: docker
      gpgcheck: no

  - name: install docker
    dnf:
      name: docker-ce
      state: present


  - name: setup docker.service
    replace:
      path: /lib/systemd/system/docker.service
      regexp: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"
      replace: "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock  --iptables=true"

  - name: setup docker registry
    copy:
      content: |
        {
          "registry-mirrors":["https://reg.timinglee.org"]
        }
      dest: /etc/docker/daemon.json

  - name: setup docker certs
    file:
      path: /etc/docker/certs.d/reg.timinglee.org
      state: directory


  - name: cp certs file
    copy:
      src: "{{item.src}}"
      dest: "{{item.dest}}"
    loop:
      - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt
        dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt

      - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt
        dest: /etc/pki/ca-trust/source/anchors/ca.crt
  
  - name: load module
    copy:
      dest: "{{item.dest}}"
      content: "{{item.value}}"
    loop:
      - dest: /etc/modules-load.d/docker_mod.conf
        value: br_netfilter
      - dest: /etc/sysctl.d/docker.conf
        value: |
          net.bridge.bridge-nf-call-iptables = 1
          net.bridge.bridge-nf-call-ip6tables = 1
          net.ipv4.ip_forward = 1

  - name: update ca cert
    shell: "update-ca-trust extract;sysctl --system"

  - name: start services
    service:
      name: "{{item}}"
      state: restarted
      enabled: yes
    loop:
      - docker
      - systemd-modules-load.service

6.利用ansible安装cri-dockerd

将安装包cri-dockerd-0.4.4.amd64.tgz放入harbor主机/home/devops/ansible目录中

[devops@harbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz
[devops@harbor ansible]$ cd cri-dockerd/
[devops@harbor cri-dockerd]$ vim cri-docker.service

在kubernetes-devops.yml后添加

[devops@harbor ansible]$ vim kubernetes-devops.yml
-------------------------------------------------------------------------------------------
- name: stup cri-dockerd
    copy:
      src: "{{item.src}}"
      dest: "{{item.dest}}"
      mode: '0755'
    loop:
      - src: ./cri-dockerd/cri-dockerd
        dest: /usr/bin/cri-dockerd

      - src: "./cri-dockerd/cri-docker.service"
        dest: "/lib/systemd/system/cri-docker.service"

      - src: "./cri-dockerd/cri-docker.socket"
        dest: "/lib/systemd/system/cri-docker.socket"

  - name: update ca cert
    shell: "update-ca-trust extract;sysctl --system;systemctl daemon-reload"

  - name: start services
    service:
      name: "{{item}}"
      state: restarted
      enabled: yes
    loop:
      - docker
      - systemd-modules-load.service
      - cri-docker

[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

7.利用ansible部署k8s软件

[devops@harbor ansible]$ mv kubernetes-devops.yml docker.yml
[devops@harbor ansible]$ vim kubernetes-devops.yml
-------------------------------------------------------------------------------------------
- name: install kubernetes from ansible
  hosts: all
  tasks:
  - name: setup k8s repo
    yum_repository:
      name: kubernetes
      description: kubernetes
      baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm
      file: kubernetes
      gpgcheck: no

  - name: install software in master
    dnf:
      name: "{{item}}"
      state: present
    loop:
      - kubeadm-1.35.7-150500.1.1
      - kubelet-1.35.7-150500.1.1
      - kubectl-1.35.7-150500.1.1
    when: inventory_hostname == "172.25.254.100"

  - name: install software in work node
    dnf:
      name: "{{item}}"
      state: present
    loop:
      - kubeadm-1.35.7-150500.1.1
      - kubelet-1.35.7-150500.1.1
    when: inventory_hostname == "172.25.254.10" or inventory_hostname == "172.25.254.20"
  
  - name: setup kubectl kubeadm complication
    lineinfile:
      path: /root/.bashrc
      line: |
        source <(kubectl completion bash)
        source <(kubeadm completion bash)
    when: inventory_hostname == "172.25.254.100"

  - name: start kubelet
    service:
      name: kubelet
      state: started
      enabled: yes
-----------------------------------------------------------------------------------------
[devops@harbor ansible]$ ansible-playbook kubernetes-devops.yml

让补齐功能生效

[root@k8s-m ~]# source  ~/.bashrc

测试补齐

8.下载kubernetes集群所需镜像

下载镜像

[root@k8s-m ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.35.3 \
--cri-socket=unix:///var/run/cri-dockerd.sock

将镜像上传至harbor

上传之前要先浏览器访问172.25.254.254

输入先前设置的账号密码(admin,lee)进行登录

创建新的项目k8s【需要开启公开】

批量贴标签并上传

[root@k8s-m ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'
[root@k8s-m ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/timinglee/{system("docker push "$0)}'

可在浏览器查看上传结果

9.在k8s-m中初始化kubernetes集群

k8s-m中

[root@k8s-m ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.timinglee.org/k8s \
--kubernetes-version v1.35.3 \
--cri-socket=unix:///var/run/cri-dockerd.sock

添加kubernets环境变量

[root@k8s-m ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" > ~/.bash_profile
[root@k8s-m ~]# source  ~/.bash_profile

添加完成后能够使用该命令

添加node节点

[root@k8s-n1 ~]# kubeadm join 172.25.254.100:6443 --token 9ravx1.taz2p7n9rdebno6u         --discovery-token-ca-cert-hash sha256:0f9078724d3761776fbf906e05ef8c11117ccaf876fe92af27bc308e686f22a9 --cri-socket=unix:///var/run/cri-dockerd.sock

[root@k8s-n2 ~]# kubeadm join 172.25.254.100:6443 --token 9ravx1.taz2p7n9rdebno6u         --discovery-token-ca-cert-hash sha256:0f9078724d3761776fbf906e05ef8c11117ccaf876fe92af27bc308e686f22a9 --cri-socket=unix:///var/run/cri-dockerd.sock

10.安装网络插件

[root@k8s-m ~]# ls
anaconda-ks.cfg                      flannel-0.28.1.tar  libcgroup-0.41-19.el8.x86_64.rpm
cri-dockerd-0.3.14-3.el8.x86_64.rpm  kube-flannel.yml
[root@master ~]# docker load  -i flannel-0.28.1.tar


[root@k8s-m ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.0-flannel1 reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
[root@k8s-m ~]# docker push  reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
The push refers to repository [reg.timinglee.org/flannel-io/flannel-cni-plugin]
2c8aa52d4746: Pushed
5aa68bbbc67e: Pushed
v1.9.0-flannel1: digest: sha256:b3d30c221113b30fea3e8a7fccb145e929b097d0319b9eeb6b5a591b10b5c671 size: 739

[root@k8s-m ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.1 reg.timinglee.org/flannel-io/flannel:v0.28.1
[root@k8s-m ~]# docker push reg.timinglee.org/flannel-io/flannel:v0.28.1
The push refers to repository [reg.timinglee.org/flannel-io/flannel]
5668da16a30b: Pushed
5f70bf18a086: Pushed
00012e17b6cc: Pushed
9738bb9596cf: Pushed
b6233bc105d7: Pushed
03767449b95b: Pushed
9a7d8cf9ff51: Pushed
13a60c10faeb: Pushed
70dc5e033175: Pushed
256f393e029f: Pushed
v0.28.1: digest: sha256:e671adbc267460164555159210066d3304a43e3b5dd85cc0b5b6ad62e83aab52 size: 2414


[root@k8s-m ~]# vim kube-flannel.yml
        image: flannel-io/flannel:v0.28.1
        image: flannel-io/flannel-cni-plugin:v1.9.0-flannel1
        image: flannel-io/flannel:v0.28.1

[root@k8s-m ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

测试