VirtualBox + Vagrant + Ansible快速搭建K8s(1.13.0)集群

VirtualBox + Vagrant + Ansible快速搭建K8s(1.13.0)集群

目标

• 一键创建虚拟服务器群组
  • 共4台虚拟机，三台建k8s，一台搭docker私有仓库
• 一键搭建K8s集群
  • 1 master
  • 2 workers

集群方案：

发行版：CentOS 7
容器运行时：Docker-18.06.1-ce
内核： 4.20.2-1.el7.elrepo.x86_64
版本：Kubernetes: 1.13.0
网络方案: Calico
kube-proxy mode: IPVS

开始前准备

• 硬件
  • 一台可联网主机(配置尽可能高)，创建虚拟机群组，本次案例采用的是macOS主机，6cpus/16G，压力不大
• 软件
  • VirtualBox: 6.0.0-127566-OSX
  • Vagrant: 2.2.3_x86_64
• 依赖资源(可先行下载下来后面需要使用)
  • docker-ce rpm包。网络环境OK的无所谓了。
    • docker-ce-18.06.1.ce-3.el7.x86_64.rpm
  • Vagrant的centos7的box，vagrant是基于box创建虚拟机，类似docker中基于image创建container。虚拟机box尽量保持一致，否则会出现各种问题！
    • CentOS-7-x86_64-Vagrant-1811_02.VirtualBox.box
  • k8s组件的rpm包及docker镜像。因大部分都在国外，不可FQ的同学可直接下载备用。这里要感谢大牛lentil1016的教程和资源!
    • k8s-v1.13.0-rpms.tgz 密码:4x77
    • k8s-repo-1.13.0 密码:aqq6

创建虚拟机群组

虚拟机的创建依靠VirtualBox和Vagrant配合完成的，安装VirtualBox和Vagrant过程不多说，直接官网走起！下面介绍一下怎么通过Vagrantfile创建虚拟机。

虚拟机配置参数

本案例我通过4台虚拟机来完成，各虚拟机配置如下：

hostname	ip	cpu	memory
k8s-master-01	10.110.111.111	2	2048
k8s-worker-01	10.110.111.112	2	2048
k8s-worker-02	10.110.111.113	2	2048
k8s-docker-register	10.110.111.120	2	1024

虚拟机创建

我们直接在一个Vagrantfile中配置所有的虚拟机：

Vagrant.configure("2") do |config|
  config.vm.define "k8s-master-01" do |master_01|
  
  end
  config.vm.define "k8s-worker-01" do |worker_01|
  
  end
  config.vm.define "k8s-worker-02" do |worker_02|
  
  end
  config.vm.define "k8s-docker-register" do |docker|
  
  end
end

其中Vagrant.configure("2")为固定格式，不需修改。后面的do和最后的end组成一个代码单元块；|config|中的config可以理解为是Vagrant.configure("2")对象的实例；config.vm.define即定义新的虚拟机“k8s-master-01”，同样以do end为单元块，所有虚拟机配置相似，下面均以‘k8s-master-01’为例来说明。

首先需要指定虚拟机的操作系统，添加上面下载的CentOS-7-x86_64-Vagrant-1811_02.VirtualBox.box到vagrant box列表中：
vagrant box add path/to/your/CentOS-7-x86_64-Vagrant-1811_02.VirtualBox.box --name centos/7

这样在Vagrantfile中通过指定box为centos/7来创建虚拟机：

Vagrant.configure("2") do |config|
  config.vm.define "k8s-master-01" do |master_01|
    master_01.vm.box = "centos/7"
  end
end

然后配置hostname、ip、cpu、memory，其中ip通过私有网络配置得到，且需配置在与你主机同一网段，否则主机无法直接通过ssh连接上虚拟机(vagrant ssh可以直接连接)：

Vagrant.configure("2") do |config|
  config.vm.define "k8s-master-01" do |master_01|
    master_01.vm.box = "centos/7"
    master_01.vm.hostname = "k8s-master-01"
  master_01.vm.network "private_network", ip: "10.110.111.111"
    master_01.vm.provider "virtualbox" do |v|
      v.memory = 2048
      v.cpus = 2
    end
  end
end

针对不同版本的centos7的vagrant box，创建出来的虚拟机基本配置环境有不同程度的差别，就本案例创建出来的虚拟机默认禁止root登陆的相关项，所以还需要修改sshd_config相关配置：

$change_sshd_config = <<-SCRIPT
echo change sshd_config to allow public key authentication & relaod sshd...
sed -i 's/\#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/\#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl reload sshd
SCRIPT

Vagrant.configure("2") do |config|

  config.vm.define "k8s-master-01" do |master_01|
    master_01.vm.box = "centos/7"
    master_01.vm.hostname = "k8s-master-01"
    master_01.vm.network "private_network", ip: "10.110.111.111"
    master_01.vm.provision "shell", inline: $change_sshd_config #调用脚本，修改sshd_config配置
    master_01.vm.provider "virtualbox" do |v|
      v.memory = 2048
      v.cpus = 2
    end
  end
end

完整的Vagrantfile如下：

$change_sshd_config = <<-SCRIPT
echo change sshd_config to allow public key authentication & relaod sshd...
sed -i 's/\#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -i 's/\#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl reload sshd
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"
  config.vm.network "public_network", bridge: "en0: Wi-Fi (Wireless)"

  config.vm.define "k8s-master-01" do |master_01|
    master_01.vm.box = "centos/7"
    master_01.vm.hostname = "k8s-master-01"
    master_01.vm.network "private_network", ip: "10.110.111.111"
    master_01.vm.provision "shell", inline: $change_sshd_config
    master_01.vm.provider "virtualbox" do |v|
      v.memory = 2048
      v.cpus = 2
    end
  end

  config.vm.define "k8s-worker-01" do |worker_01|
    worker_01.vm.box = "centos/7"
    worker_01.vm.hostname = "k8s-worker-01"
    worker_01.vm.network "private_network", ip: "10.110.111.112"
    worker_01.vm.provision "shell", inline: $change_sshd_config
    worker_01.vm.provider "virtualbox" do |v|
      v.memory = 2048
      v.cpus = 2
    end
  end

  config.vm.define "k8s-worker-02" do |worker_02|
    worker_02.vm.box = "centos/7"
    worker_02.vm.hostname = "k8s-worker-02"
    worker_02.vm.network "private_network", ip: "10.110.111.113"
    worker_02.vm.provision "shell", inline: $change_sshd_config
    worker_02.vm.provider "virtualbox" do |v|
      v.memory = 2048
      v.cpus = 2
    end
  end

  config.vm.define "k8s-docker-register" do |docker|
    docker.vm.box = "centos/7"
    docker.vm.hostname = "k8s-docker-register"
    docker.vm.network "private_network", ip: "10.110.111.120"
    docker.vm.provision "shell", inline: $change_sshd_config
    docker.vm.provider "virtualbox" do |v|
      v.memory = 1024
      v.cpus = 2
    end
  end
end

其中config.vm.network "public_network", bridge: "en0: Wi-Fi (Wireless)"添加了公共网络，用于外网访问虚拟机，可根据需求自行添加。
自此，创建虚拟机群组的Vagrantfile完成，添加到工作目录～/vagrant/vm/k8s，执行vagrant up创建虚拟机：

macbook-pro:k8s jason$ vagrant up
Bringing machine 'k8s-master-01' up with 'virtualbox' provider...
Bringing machine 'k8s-worker-01' up with 'virtualbox' provider...
Bringing machine 'k8s-worker-02' up with 'virtualbox' provider...
Bringing machine 'k8s-docker-register' up with 'virtualbox' provider...
==> k8s-master-01: Importing base box 'centos/7'...
==> k8s-master-01: Matching MAC address for NAT networking...
==> k8s-master-01: Setting the name of the VM: k8s_k8s-master-01_1547193895558_14331
==> k8s-master-01: Fixed port collision for 22 => 2222. Now on port 2200.
==> k8s-master-01: Clearing any previously set network interfaces...
==> k8s-master-01: Preparing network interfaces based on configuration...
    k8s-master-01: Adapter 1: nat
    k8s-master-01: Adapter 2: bridged
    k8s-master-01: Adapter 3: hostonly
==> k8s-master-01: Forwarding ports...
    k8s-master-01: 22 (guest) => 2200 (host) (adapter 1)
==> k8s-master-01: Running 'pre-boot' VM customizations...
==> k8s-master-01: Booting VM...
==> k8s-master-01: Waiting for machine to boot. This may take a few minutes...
    k8s-master-01: SSH address: 127.0.0.1:2200
    k8s-master-01: SSH username: vagrant
    k8s-master-01: SSH auth method: private key
    k8s-master-01: 
    k8s-master-01: Vagrant insecure key detected. Vagrant will automatically replace