前言
修改K8S版本bug
Kubernetes v1.20 (opens new window)开始，默认删除了 metadata.selfLink 字段，
然而，部分应用仍然依赖于这个字段，例如 nfs-client-provisioner
通过配置 apiserver 启动参数中的 –feature-gates 中的 RemoveSelfLink=false，
可以重新启用 metadata.selfLink 字段
1.19版本之上需要修改api
修改/etc/kubernetes/manifests/kube-apiserver.yaml 文件
添加添加- --feature-gates=RemoveSelfLink=false
意思是：运行nfs的接口进行认证自动调度
Milvus集群1.1.1版本
共享存储storageclass
git clone https://github.com/helm/charts.git
这里是国外的源，下载要很久，我已经下载好打包放在当前目录：charts.tar.gz
[root@master001 opt]# cd charts/stable/nfs-client-provisioner
[root@master001 opt]# vim values.yaml
13 nfs:
14 server: 20.0.0.25
15 path: /data/nfs
16 mountOptions:
17 - rw
server 参数改为共享存储服务器的 IP 地址，将 path 参数改为共享存储目录。
此外,由 StorageClass 动态创建的 PersistentVolume 将使用 mountOptions 字段指定的挂载选项
[root@master001 nfs-client-provisioner]# helm install nfs-client .
WARNING: This chart is deprecated
NAME: nfs-client
LAST DEPLOYED: Thu Mar 10 16:27:16 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
[root@master001 ~]# kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client cluster.local/nfs-client-nfs-client-provisioner Delete Immediate true <invalid>
[root@master001 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-nfs-client-provisioner-6d4cdd4b69-2vnpp 1/1 Running 0 4m21s
helm部署Milvus集群
helm repo add stable https://charts.helm.sh/stable
helm repo update
随后将milvus-helm-1.1.zip 拖进来解压
[root@master001 milvus]# cd milvus-helm-1.1/charts/milvus/
##将所有的storageClass与accessModes变成下面的：
[root@k8s-master01 milvus]# vim values.yaml
storageClass: nfs-client
accessModes: ReadWriteMany
[root@master001 milvus]# helm dep update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading mysql from repo https://charts.helm.sh/stable
Deleting outdated charts
#以下配置酌情使用
helm install
--set cluster.enabled=true ##开启集群模式
--set persistence.enabled=true
--set mishards.replicas=2 #2个分片中间件、2个读节点、1个写节点
--set readonly.replicas=2
--set mysql.enabled=false #关闭集群内部mysql，使用外部的
--set externalMysql.enabled=true
--set externalMysql.ip=193.168.57.111
--set externalMysql.port=3306
--set externalMysql.user=milvus
--set externalMysql.password=Milvus
--set externalMysql.database=milvus
--set cache.cpuCacheCapacity=8 #16GB内存的读节点和8GB内存的写节点
--set readonly.cache.cpuCacheCapacity=16
--set gpu.enabled=true #允许写节点使用GPU资源
--set readonly.gpu.enabled=false # 禁止读节点使用GPU资源
milvus-release .
#我这里使用的外部mysql集群，所以不存在mysql的pod
[root@k8s-master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
call-12345-client-648c774d65-49nhb 1/1 Running 0 2d21h
call-12345-client-648c774d65-zfhh5 1/1 Running 0 2d21h
call-12345-server-84b7f58c54-2xps6 1/1 Running 11 5d20h
call-12345-server-84b7f58c54-prfk8 1/1 Running 1 5d5h
milvus-em-79f7cbbcd7-vmcbh 1/1 Running 2 5d22h
milvus-release-mishards-767dc476bf-fxz8v 1/1 Running 2 9d
milvus-release-mishards-767dc476bf-xpgf4 1/1 Running 2 9d
milvus-release-readonly-6cb6c5bcd9-9m4c5 1/1 Running 2 9d
milvus-release-readonly-6cb6c5bcd9-mzrh6 1/1 Running 2 9d
milvus-release-writable-7776777fcc-dsdmv 1/1 Running 2 9d
nfs-client-nfs-client-provisioner-6554ff4498-6blmv 1/1 Running 2 9d
Milvus分布式2.0x版本
创建共享存储
[root@k8s-master01 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs (default) nfs-nfs Delete Immediate false 82m
[root@k8s-master01 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
eip-nfs-nfs-6f96c578d9-qbqc5 1/1 Running 0 85m
安装Milvus分布式2.0x
使用 Kubernetes 包管理工具 Helm 添加 Milvus chart 仓库：
$ helm repo add milvus https://milvus-io.github.io/milvus-helm/
将 Milvus chart 更新至最新版本：
$ helm repo update
安装 Milvus Helm chart：
设置发布命名（release name）以标记或追踪该 chart 部署。
本教程使用 my-release 作为 release name。如需使用不同的 release name, 请在以下命令中修改相应的 release name。
Release name 中只能包含字母、数字以及下划线，不能包含点。
安装 Milvus 分布式版：
$ helm install my-release milvus/milvus
[root@k8s-master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-release-etcd-0 1/1 Running 0 84m
my-release-etcd-1 1/1 Running 0 84m
my-release-etcd-2 1/1 Running 0 84m
my-release-milvus-datacoord-7cdb774fc4-htkxp 1/1 Running 0 84m
my-release-milvus-datanode-5bc7c5847-hcbpt 1/1 Running 0 84m
my-release-milvus-indexcoord-5496c99f-l448h 1/1 Running 0 84m
my-release-milvus-indexnode-554b87cbb6-6bl2x 1/1 Running 0 84m
my-release-milvus-proxy-598bb877c-rfvz2 1/1 Running 0 84m
my-release-milvus-querycoord-78f8f7ffb-xzmkw 1/1 Running 0 84m
my-release-milvus-querynode-855cddd5cb-mfh64 1/1 Running 0 84m
my-release-milvus-rootcoord-858f89bf54-4fhk6 1/1 Running 0 84m
my-release-minio-0 1/1 Running 0 84m
my-release-minio-1 1/1 Running 0 84m
my-release-minio-2 1/1 Running 0 84m
my-release-minio-3 1/1 Running 0 84m
my-release-pulsar-bookie-0 1/1 Running 0 84m
my-release-pulsar-bookie-1 1/1 Running 0 84m
my-release-pulsar-bookie-2 1/1 Running 0 84m
my-release-pulsar-bookie-init-d86jg 0/1 Completed 0 84m
my-release-pulsar-broker-0 1/1 Running 0 84m
my-release-pulsar-proxy-0 1/1 Running 0 84m
my-release-pulsar-pulsar-init-99l6j 0/1 Completed 0 84m
my-release-pulsar-recovery-0 1/1 Running 0 84m
my-release-pulsar-zookeeper-0 1/1 Running 0 84m
my-release-pulsar-zookeeper-1 1/1 Running 0 83m
my-release-pulsar-zookeeper-2 1/1 Running 0 83m
查看一下20.0.0.21下的/data
[root@k8s-master01 data]# ll
total 0
drwxrwxrwx 3 root root 18 May 10 15:05 default-data-my-release-etcd-0-pvc-5b8cccb3-1f3d-4f0f-98ea-6d34f2eeb7b7
drwxrwxrwx 3 root root 18 May 10 15:05 default-data-my-release-etcd-1-pvc-0a65c9db-2381-4455-92de-38f133dec0ec
drwxrwxrwx 3 root root 18 May 10 15:05 default-data-my-release-etcd-2-pvc-22419067-0f52-4618-abbf-a8f55007d372
drwxrwxrwx 4 root root 45 May 10 15:06 default-export-my-release-minio-0-pvc-cecc674b-8522-4969-a333-74764b6e78e5
drwxrwxrwx 4 root root 109 May 10 15:06 default-export-my-release-minio-1-pvc-e739da8a-4601-4e03-8f34-b1af0aa53e6a
drwxrwxrwx 4 root root 45 May 10 15:06 default-export-my-release-minio-2-pvc-5acfcd5a-db35-4ab5-a275-6d9c0fac9017
drwxrwxrwx 4 root root 109 May 10 15:06 default-export-my-release-minio-3-pvc-785d524e-fbc2-45ec-a27c-46eb4b24c02f
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-journal-my-release-pulsar-bookie-0-pvc-372e47c1-f2b5-4a5b-8c06-5d7e3899c626
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-journal-my-release-pulsar-bookie-1-pvc-919b786b-4ae9-4ac1-9b08-0c87c9b1dedf
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-journal-my-release-pulsar-bookie-2-pvc-05009e56-b712-4395-9fbb-c7c4828264bc
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-ledgers-my-release-pulsar-bookie-0-pvc-9afbcc72-ba70-4b43-a4ed-2f24b70b6417
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-ledgers-my-release-pulsar-bookie-1-pvc-5c9ce2b8-f879-4ab6-8742-1df4a39dff8c
drwxrwxrwx 3 root root 21 May 10 15:06 default-my-release-pulsar-bookie-ledgers-my-release-pulsar-bookie-2-pvc-c0e0a9f0-41c6-499a-8c41-64fbdc05e921
drwxrwxrwx 3 root root 23 May 10 15:05 default-my-release-pulsar-zookeeper-data-my-release-pulsar-zookeeper-0-pvc-49d11f41-4b48-43d7-a473-6f65730b3ccb
drwxrwxrwx 3 root root 23 May 10 15:05 default-my-release-pulsar-zookeeper-data-my-release-pulsar-zookeeper-1-pvc-564998f2-e69a-4980-aa66-28573658e579
drwxrwxrwx 3 root root 23 May 10 15:06 default-my-release-pulsar-zookeeper-data-my-release-pulsar-zookeeper-2-pvc-2bfc6e3a-3540-41d5-9b93-1b0701cc5cfb
更改my-release-milvus的svc为nodePort模式
[root@k8s-master01 nfs]# kubectl edit svc my-release-milvus
....... 在末尾处
selector:
app.kubernetes.io/instance: my-release
app.kubernetes.io/name: milvus
component: proxy
sessionAffinity: None
type: NodePort #修改为NodePort
status:
loadBalancer: {}
这样我们就可以通过集群的vip+svc暴露的端口来访问milvus了
[root@k8s-master01 nfs]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-release-milvus NodePort 10.102.61.43 <none> 19530:31063/TCP,9091:30898/TCP 85m
安装attu
#执行后发现日志全部在桌面上，所以这里我们加一个-d
docker run -d -p 8000:3000 -e HOST_URL=http://本机:8000 -e MILVUS_URL=K8S集群的vip:svc暴露的端口31063 zilliz/attu:latest
指定版本安装
比如我想安装2.0.2版本，需要指定对应的Chart Version
helm install my-release milvus/milvus --version 3.0.29
下载chart包到本地，指定版本
helm pull milvus/milvus --version 3.0.29
helm milvus更多配置
升级文档参考
https://milvus.io/docs/v2.1.x/upgrade.md#Upgrade-Milvus-cluster
2191
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
