《kubernetes-1.8.0》09-addon-kube-prometheus
《kubernetes 1.8.0 测试环境安装部署》
时间:2017-11-23
一、环境准备
clone kube-prometheus 项目:
git clone https://github.com/coreos/prometheus-operator.git
指定kubeconfig目录位置(本例使用缺省,如果配置文件存放位置不是~/.kube/config需指定该变量):
export KUBECONFIG=<path> # defaults to "~/.kube/config"
二、安装kube-prometheus
运行kube-prometheus部署脚本:
$ cd prometheus-operator/contrib/kube-prometheus/
$ hack/cluster-monitoring/deploy
...
namespace "monitoring" created
clusterrolebinding "prometheus-operator" created
clusterrole "prometheus-operator" created
serviceaccount "prometheus-operator" created
service "prometheus-operator" created
deployment "prometheus-operator" created
Waiting for Operator to register custom resource definitions....................
daemonset "node-exporter" created
service "node-exporter" created
clusterrolebinding "kube-state-metrics" created
clusterrole "kube-state-metrics" created
deployment "kube-state-metrics" created
rolebinding "kube-state-metrics" created
role "kube-state-metrics-resizer" created
serviceaccount "kube-state-metrics" created
service "kube-state-metrics" created
secret "grafana-credentials" created
secret "grafana-credentials" configured
configmap "grafana-dashboards-0" created
deployment "grafana" created
service "grafana" created
servicemonitor "kube-state-metrics" created
servicemonitor "prometheus-operator" created
prometheus "k8s" created
servicemonitor "kubelet" created
servicemonitor "kube-controller-manager" created
serviceaccount "prometheus-k8s" created
servicemonitor "prometheus" created
servicemonitor "kube-apiserver" created
servicemonitor "alertmanager" created
servicemonitor "kube-scheduler" created
configmap "prometheus-k8s-rules" created
servicemonitor "node-exporter" created
service "prometheus-k8s" created
role "prometheus-k8s" created
role "prometheus-k8s" created
role "prometheus-k8s" created
clusterrole "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
clusterrolebinding "prometheus-k8s" created
secret "alertmanager-main" created
service "alertmanager-main" created
alertmanager "main" created
创建pod时间较长,镜像需要下比较久,其中 addon-resizer:1.0
在之前的镜像包中提供。
一个小插曲:
加载完毕后,30900端口访问失败,怀疑prometheus核心组件启动失败,但检查pod又都是Running。
查看日志后发现:
kube-controller-manage
报错:
Event(v1.ObjectReference{Kind:”StatefulSet”, Namespace:”monitoring”, Name:”prometheus-k8s”, UID:”2fbbb615-d117-11e7-917e-005056bc52e5”, APIVersion:”apps”, ResourceVersion:”256502”, FieldPath:”“}): type: ‘Warning’ reason: ‘FailedCreate’ create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s failed error: pods “prometheus-k8s-0” is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden
检查statefulset启动情况:
发现prometheus-k8s
这个statefulset没有成功启动:
用describe查看:
排查半天,发觉可能是apiserver 的控制问题,检查apiserver配置文件:
去除:KUBE_ADMISSION_CONTROL
中的SecurityContextDeny
部分,分别重启apiserver。
再次检查statefulset和pod的情况:
[root@node-131 kube-prometheus]# kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 16m
alertmanager-main-1 2/2 Running 0 15m
alertmanager-main-2 2/2 Running 0 15m
grafana-7d966ff57-twzx9 2/2 Running 0 17m
kube-state-metrics-b5f8d6c5-zvgh5 2/2 Running 0 17m
node-exporter-7zx6s 1/1 Running 0 17m
node-exporter-b88t5 1/1 Running 0 17m
node-exporter-dhx7x 1/1 Running 0 17m
node-exporter-wnhvd 1/1 Running 0 17m
prometheus-k8s-0 2/2 Running 0 16m
prometheus-k8s-1 2/2 Running 0 16m
prometheus-operator-66578f9cd9-wstrp 1/1 Running 0 17m
[root@node-131 kube-prometheus]# kubectl get statefulset -n monitoring
NAME DESIRED CURRENT AGE
alertmanager-main 3 3 16m
prometheus-k8s 2 2 16m
SecurityContextDeny
:具体的作用需查看官档。
三、测试访问:
访问prometheus(30900):
访问alertmanager(30903):
访问grafana(30902):
至此kube-prometheus框架基本搭建完成,后续关于prometheus具体操作另写:
本系列其他内容:
参考文档:
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus