prometheus-operator部署

关注

原创最新推荐文章于 2026-06-21 19:47:51 发布 383 阅读 AI 写同款· GEO 优化 ›

Prometheus学习整理-Prometheus-operator 阅读详情

一、前言

prometheus-operator相比较传统的prometheus主要是使用CRD的方式实现部署的自动化，还可以自动发现监控的节点、自动发现监控规则并更新，不需要像传统的方式一样需要手动执行命令热加载，本篇文章使用prometheus-operator的方式部署prometheus、alertmange、servicemonitor、prometheusrule并且使用prometheusalert组件告警到企业微信机器人，本人使用的是k8s 1.23.0版本

二、实施

创建命名空间

kubectl create ns prometheus

创建私有仓库拉取凭据

 kubectl create secret docker-registry monitor-secret --namespace=prometheus --docker-server=https://xxx.xxx.com --docker-username=xxx --docker-password=xxx

部署prometheus-operator

在github上下载yaml部署文件,根据自己的k8s集群版本选择对应的operator版本

参考连接：https://github.com/prometheus-operator/prometheus-operator/blob/main/bundle.yaml

下载完成后可以自定义镜像、以及命名空间

#需要更改namespace的位置
kind: ClusterRoleBinding
  namespace: prometheus
kind: Deployment
  namespace: prometheus
kind: ServiceAccount
  namespace: prometheus
kind: Service
  namespace: prometheus

#需要更改image的位置
kind: Deployment
  image: xxx.xxx.com/monitor/prometheus-operator:v0.66.0
  - --prometheus-config-reloader=xxx.xxx.com/monitor/prometheus-config-reloader:v0.66.0

#更改私有仓库镜像后需要增加拉取镜像凭据配置

  imagePullSecrets:
  - name: monitor-secret

更改完成后直接执行以下命令部署

#需要使用create命令，使用apply命令会报错
kubectl create -f bundle.yaml

部署完成后可以看到创建的operator

kubectl get pod -n prometheus

部署prometheus，prometheus通过创建yaml文件，然后由rbc识别自动部署

deployment.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: prometheus
spec:
  replicas: 1
  # 1. 使用私有仓库镜像
  image: harbor.ylbeijing.com/monitor/prometheus:latest  # 替换为你的私有仓库地址
  imagePullPolicy: IfNotPresent
  
  # 2. 私有仓库认证（如果需要）
  imagePullSecrets:
  - name: monitor-secret  # 需要提前创建 docker-registry secret
  
  #配置altermanage告警
  alerting:
    alertmanagers:
    - namespace: prometheus
      name: alertmanager-operated
      port: 9093

  #用于发现k8s集群外的监控服务
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

  # 3. 使用 PVC 持久化存储
  storage:
    volumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteMany"]
        resources:
          requests:
            storage: 30Gi
        # 如果使用方式B的本地存储，取消下面的注释
        storageClassName: nfs

  #使用root模式,因为nfs配置的目录权限是root,用其他角色权限不够报错
  securityContext:
    fsGroup: 0
    runAsUser: 0
    runAsNonRoot: false
 
  # 4. 基础配置
  serviceAccountName: prometheus
  replicas: 1
  retention: 30d  # 数据保留30天
  retentionSize: 30GB  # 可选：限制存储大小
  
  # 5. ServiceMonitor 选择器
  serviceMonitorSelector:
    matchLabels: {}
  serviceMonitorNamespaceSelector: {}
  
  #prometheus告警规则自动发现
  ruleSelector:
    matchLabels: {}
  ruleNamespaceSelector: {}

  # 6. 资源配置
  resources:
    requests:
      memory: 1Gi
      cpu: 1000m
    limits:
      memory: 2Gi
      cpu: 2000m
  
  # 7. 节点亲和性（使用本地存储时必须指定）
  # nodeSelector:
  #   kubernetes.io/hostname: your-node-name

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: prometheus

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "prometheus-data-pv"
  labels:
    name: prometheus-data-pv
    release: stable
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteMany            #配置读写方式
  persistentVolumeReclaimPolicy: Retain  #配置回收策略为Retain，既删除pv保留数据
  storageClassName: nfs       #使用nfs存储
  nfs:         #配置nfs路径信息
    path: /xxx/xxx/xxxx
    server: xx.xx.xx.xx

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data-pvc
  namespace: prometheus
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      name: prometheus-data-pv
      release: stable
  storageClassName: nfs

service.yaml

kind: Service
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app.kubernetes.io/name: prometheus
  name: prometheus-service
  namespace: prometheus
spec:
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30090
  selector:
    app.kubernetes.io/name: prometheus
  type: NodePort

部署servicemonitor 用于pod自动发现

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: servicemonitor          # 给这个监控任务起个名字
  namespace: prometheus          # 建议与 Prometheus 实例所在命名空间一致
  labels:
    # 【关键】这里的标签需要能被你的 Prometheus 实例的 serviceMonitorSelector 选中
    # 你可以通过以下命令查看你的 Prometheus 实例需要什么标签
    # kubectl describe prometheus prometheus -n prometheus | grep "Service Monitor Selector"
    release: prometheus          # 这是一个常见示例，请根据你的实际配置修改
spec:
  # 选择要监控的 Service
  selector:
    matchLabels:
      #app: my-app               # 匹配 labels 为 app: my-app 的 Service
      monitoring: "enabled"  # 所有服务只要打了这个标签就被监控
  # 配置采集端点
  endpoints:
  - port: metrics               # 指定 Service 上定义的端口名称，必须与要自动发现的svc端口名称对应不然识别不到
    interval: 30s               # 每30秒采集一次
  # 指定要监控的 Service 所在的命名空间
  namespaceSelector:
    #matchNames:
    #- my-namespace              # 替换为你的应用所在的命名空间
    any: true  # 监控所有命名空间

部署prometheusrule用于prometheus自动发现告警规则

以下规则仅供参考，需要其他的告警规则可以参考以下网站

prometheus告警规则网站：Awesome Prometheus Alerts | Copy-pasteable Prometheus alerting rules

prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prometheusrule
  namespace: prometheus            # 通常放在和 Prometheus 相同的命名空间
  labels:
    # 同样，这里的 labels 需要被你的 Prometheus 实例的 ruleSelector 选中
    release: prometheus
spec:
  groups:
  - name: nodealerts
    rules:
    - alert: 内存使用率过高
      expr: 100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
      for: 3m
      labels:
        severity: critical
      annotations:
        summary: "{{ $labels.instance }} 内存使用率过高, 请尽快处理！"
        description: "{{ $labels.instance }}内存使用率超过85%,当前使用率{{ $value }}%."
    - alert: 服务器宕机
      expr: up == 0
      for: 1s
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.instance}} 服务器宕机, 请尽快处理!"
        description: "{{$labels.instance}} 服务器延时超过3分钟,当前状态{{ $value }}. "
    - alert: CPU高负荷
      expr: 100 - (avg by (instance,job)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.instance}} CPU使用率过高,请尽快处理！"
        description: "{{$labels.instance}} CPU使用大于80%,当前使用率{{ $value }}%. "
      
    - alert: 磁盘IO性能
      expr: avg(irate(node_disk_io_time_seconds_total[1m])) by(instance,job)* 100 > 80
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.instance}} 流入磁盘IO使用率过高,请尽快处理！"
        description: "{{$labels.instance}} 流入磁盘IO大于80%,当前使用率{{ $value }}%."
 
 
    - alert: 网络流入
      expr: ((sum(rate (node_network_receive_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance,job)) / 100) > 102400
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.instance}} 流入网络带宽过高，请尽快处理！"
        description: "{{$labels.instance}} 流入网络带宽持续5分钟高于100M. RX带宽使用量{{$value}}."
 
    - alert: 网络流出
      expr: ((sum(rate (node_network_transmit_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance,job)) / 100) > 102400
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.instance}} 流出网络带宽过高,请尽快处理！"
        description: "{{$labels.instance}} 流出网络带宽持续5分钟高于100M. RX带宽使用量{$value}}."
  
    - alert: TCP连接数
      expr: node_netstat_Tcp_CurrEstab > 10000
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: " TCP_ESTABLISHED过高！"
        description: "{{$labels.instance}} TCP_ESTABLISHED大于100%,当前使用率{{ $value }}%."
 
    - alert: 磁盘容量
      expr: 100-(node_filesystem_free_bytes{fstype=~"ext4|xfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs"}*100) > 80
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "{{$labels.mountpoint}} 磁盘分区使用率过高，请尽快处理！"
        description: "{{$labels.instance}} 磁盘分区使用大于80%，当前使用率{{ $value }}%."
  - name: winalerts
    rules:
    - alert: WindowsServerServiceStatus
      expr: windows_service_status{status="ok"} != 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Windows Server service Status (instance {{ $labels.instance }})
        description: "Windows Service state is not OK\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: WindowsServerCPUUsage
      expr: 100 - (avg by (instance) (rate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 80
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: Windows Server CPU Usage (instance {{ $labels.instance }})
        description: "CPU Usage is more than 80%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: WindowsServerMemoryUsage
      expr: 100 - ((windows_os_physical_memory_free_bytes / windows_cs_physical_memory_bytes) * 100) > 80
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Windows Server memory Usage (instance {{ $labels.instance }})
        description: "Memory usage is more than 80%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: WindowsServerDiskSpaceUsage
      expr: 100 - 100 * (windows_logical_disk_free_bytes / windows_logical_disk_size_bytes) > 80 and windows_logical_disk_size_bytes > 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: Windows Server disk Space Usage (instance {{ $labels.instance }})
        description: "Disk usage is more than 80%\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - name: k8salerts
    rules:
    - alert: KubernetesNodeNotReady
      expr: kube_node_status_condition{condition="Ready",status="true"} == 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: Kubernetes Node not ready (instance {{ $labels.instance }})
        description: "Node {{ $labels.node }} has been unready for a long time\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesNodeSchedulingDisabled
      expr: kube_node_spec_taint{key="node.kubernetes.io/unschedulable"} == 1
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: Kubernetes Node scheduling disabled (instance {{ $labels.instance }})
        description: "Node {{ $labels.node }} has been marked as unschedulable for more than 30 minutes.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesStatefulSetDown
      expr: kube_statefulset_replicas != kube_statefulset_status_replicas_ready > 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Kubernetes StatefulSet down (instance {{ $labels.instance }})
        description: "StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} went down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesPodNotHealthy
      expr: sum by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"}) > 0
      for: 15m
      labels:
        severity: critical
      annotations:
        summary: Kubernetes Pod not healthy (instance {{ $labels.instance }})
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-running state for longer than 15 minutes.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesPodCrashLooping
      expr: increase(kube_pod_container_status_restarts_total[1m]) > 3
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Kubernetes pod crash looping (instance {{ $labels.instance }})
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesDeploymentReplicasMismatch
      expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: Kubernetes Deployment replicas mismatch (instance {{ $labels.instance }})
        description: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replicas mismatch\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesStatefulSetReplicasMismatch
      expr: kube_statefulset_status_replicas_ready != kube_statefulset_status_replicas
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: Kubernetes StatefulSet replicas mismatch (instance {{ $labels.instance }})
        description: "StatefulSet does not match the expected number of replicas.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: KubernetesClientCertificateExpiresNextWeek
      expr: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 7*24*60*60
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: Kubernetes client certificate expires next week (instance {{ $labels.instance }})
        description: "A client certificate used to authenticate to the apiserver is expiring next week.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - name: mysqlalerts
    rules:
    - alert: MySQLDown
      expr: mysql_up == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: MySQL down (instance {{ $labels.instance }})
        description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: MySQLTooManyConnections(>80%)
      expr: max_over_time(mysql_global_status_threads_connected[1m]) / mysql_global_variables_max_connections * 100 > 80 and mysql_global_variables_max_connections > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }})
        description: "More than 80% of MySQL connections are in use on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: MySQLHighThreadsRunning
      expr: max_over_time(mysql_global_status_threads_running[1m]) / mysql_global_variables_max_connections * 100 > 60 and mysql_global_variables_max_connections > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL high threads running (instance {{ $labels.instance }})
        description: "More than 60% of MySQL connections are in running state on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: MySQLSlowQueries
      expr: delta(mysql_global_status_slow_queries[1m]) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL slow queries (instance {{ $labels.instance }})
        description: "MySQL server has some new slow queries ({{ $value }} in the last minute).\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  - name: blackboxalerts
    rules:
    - alert: BlackboxProbeFailed
      expr: probe_success == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Blackbox probe failed (instance {{ $labels.instance }})
        description: "Probe failed\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: BlackboxProbeHTTPFailure
      expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Blackbox probe HTTP failure (instance {{ $labels.instance }})
        description: "HTTP status code is not 200-399\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    - alert: BlackboxSSLCertificateWillExpireVerySoon
      expr: 0 <= round((last_over_time(probe_ssl_earliest_cert_expiry[10m]) - time()) / 86400, 0.1) < 3
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Blackbox SSL certificate will expire very soon (instance {{ $labels.instance }})
        description: "SSL certificate expires in less than 3 days\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

部署用于发现k8s集群外的监控数据接口服务

vm-scrape-secret.yaml

#用于发现k8s集群外的监控数据接口
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
  namespace: prometheus

stringData:
  prometheus-additional.yaml: |
    - job_name: 'vm-node-exporter'   #用于发现集群外的node_exporter数据接口

      static_configs:
      - targets:
        - xxx.x.xxx.xxx:9100
        - xxx.xxx.xxx.xxx:9100
        - xxx.xxx.xxx.xxx:9101
    
    - job_name: 'blackbox-http'    #用于发现集群外的blackbox_exporter数据接口

      metrics_path: /probe

      params:
        module:
        - http_2xx

      static_configs:
      - targets:
        - https://xxx.xxxx.com
        - http://xxx.xxxx.com

      relabel_configs:

      - source_labels:
        - __address__
        target_label: __param_target

      - source_labels:
        - __param_target
        target_label: instance

      - target_label: __address__
        replacement: blackbox-exporter.prometheus.svc:9115

执行以下命令部署以上yaml文件的服务

kubectl apply -f ./

部署alertmange服务用于接收prometheus的告警

deployment.yaml

#指定镜像，用rbc自动部署
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: alertmanager
  namespace: prometheus

spec:
  replicas: 1
  image: xxx/xxx/alertmanager:latest  # 替换为你的私有仓库地址
  imagePullPolicy: IfNotPresent
  
  # 2. 私有仓库认证（如果需要）
  imagePullSecrets:
  - name: monitor-secret  # 需要提前创建 docker-registry secret

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: svc-alertmanager
  namespace: prometheus
spec:
  type: NodePort
  ports:
  - port: 9093
    protocol: TCP
    targetPort: 9093
    nodePort: 30011
  selector:
    app.kubernetes.io/name: alertmanager

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-alertmanager
  namespace: prometheus

stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 1m
    route:
      group_by: ['instance']
      group_wait: 30s    #告警等待发送时间
      group_interval: 5m
      repeat_interval: 15m   #重复告警的再次发送时间
      receiver: prometheusalert       #调用的告警组件，下面配置的组件名称
    receivers:
    - name: 'prometheusalert'
      webhook_configs:
      - "url": "http://prometheus-alert-svc.prometheus:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxx"
#以上url使用的是prometheusalert,其中使用企业微信机器人的模式发送告警
        "send_resolved": true # 告警结束后是否通知已恢复

执行以下命令部署以上yaml文件的服务

kubectl apply -f ./

部署prometheusalert服务用于接收alermange的告警然后发送到企业微信机器人

部署参考：https://feiyu563.gitbook.io/prometheusalert/system/system-prometheus

配置文件参考：https://github.com/feiyu563/PrometheusAlert/blob/master/conf/app-example.conf

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-alert
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-alert
  template:
    metadata:
      labels:
        app: prometheus-alert
    spec:
      containers:
      - name: prometheus-alert
        image: xxxx/xxx/prometheus-alert:v4.9.2
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /app/conf/app.conf
          subPath: app.conf
      volumes:
      - name: config
        configMap:
          name: prometheus-alert-conf
      imagePullSecrets:
      - name: monitor-secret

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-alert-conf
  namespace: prometheus # 根据你的命名空间调整
data:
  app.conf: |
    #登录用户名
    login_user=prometheusalert
    #登录密码
    login_password=prometheusalert
    #监听地址
    httpaddr = "0.0.0.0"
    #监听端口
    httpport = 8080
    runmode = dev
    copyrequestbody = true
    title=PrometheusAlert
    logtype=file
    logpath=logs/prometheusalertcenter.log
    prometheus_cst_time=0
    db_driver=sqlite3
    AlertRecord=1
    RecordLive=1
    RecordLiveDay=30
    maxIdleConns=100
    open-hotreload=0
    open-weixin=1
    wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxx
    open-alertgroup=0

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: prometheus-alert-svc
  namespace: prometheus
spec:
  type: NodePort
  selector:
    app: prometheus-alert
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 30012
    protocol: TCP

执行以下命令部署以上yaml文件的服务

kubectl apply -f ./

部署node_exporter服务监控主机数据

监控k8s集群主机

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: prometheus
  labels:
    app: node-exporter

spec:
  selector:
    matchLabels:
      app: node-exporter

  template:
    metadata:
      labels:
        app: node-exporter
        monitoring: "enabled"

    spec:
      hostNetwork: true
      hostPID: true
      hostIPC: true

      tolerations:
      - operator: Exists

      containers:
      - name: node-exporter
        image: harbor.ylbeijing.com/monitor/node-exporter:v1.7.0

        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root

        ports:
        - name: metrics
          containerPort: 9100

        securityContext:
          privileged: true

        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true

        - name: sys
          mountPath: /host/sys
          readOnly: true

        - name: root
          mountPath: /host/root
          readOnly: true

      volumes:
      - name: proc
        hostPath:
          path: /proc

      - name: sys
        hostPath:
          path: /sys

      - name: root
        hostPath:
          path: /
      imagePullSecrets:
      - name: monitor-secret

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: prometheus

  labels:
    monitoring: "enabled"   #用于servicemonitor自动发现

spec:
  selector:
    app: node-exporter

  ports:
  - name: metrics
    port: 9100
    targetPort: 9100

执行以下命令部署以上yaml文件的服务

kubectl apply -f ./

监控虚拟机，使用脚本部署(我这里把自动下载安装包注释掉了，需要自行下载上传)

node_exporter.sh

#!/bin/bash
#mkdir /opt/node_exporter && cd /opt/node_exporter
# 下载并解压
#wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# 创建 systemd 服务 (可选，用于开机自启)
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

# 启动服务
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

部署完成后即可在prometheus中看到监控的虚拟机

部署kube-state-metrics服务监控k8s集群数据

部署文件参考：https://github.com/kubernetes/kube-state-metrics/tree/v2.7.0/examples/standard

重点需要更改service.yaml使得servicemonitor可以自动发现

service.yam

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.7.0
    monitoring: enabled   #增加servicemonitor自动发现的标签
  name: kube-state-metrics
  namespace: prometheus
spec:
  clusterIP: None
  ports:
  - name: metrics
    port: 8080
    targetPort: 8080
  - name: telemetry
    port: 8081
    targetPort: 8081
  selector:
    app.kubernetes.io/name: kube-state-metrics

部署mysql-exporter服务监控mysql数据

需要提前在数据库创建用户

CREATE USER 'exporter'@'%' IDENTIFIED BY '你的密码' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
FLUSH PRIVILEGES;

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql-exporter
  template:
    metadata:
      labels:
        app: mysql-exporter
    spec:
      containers:
      - name: mysql-exporter
        image: harbor.ylbeijing.com/monitor/mysqld-exporter:latest  # 建议使用较新版本
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9104
          name: metrics
        args:
        - "--config.my-cnf=/etc/mysql/.my.cnf"

        volumeMounts:
        - name: mysql-config
          mountPath: /etc/mysql
          readOnly: true

      volumes:
      - name: mysql-config
        secret:
          secretName: mysql-exporter-secret
      imagePullSecrets:
      - name: monitor-secret
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-exporter
  namespace: prometheus
  labels:
    app: mysql-exporter
    monitoring: enabled
spec:
  selector:
    app: mysql-exporter
  ports:
  - protocol: TCP
    port: 9104
    targetPort: 9104
    name: metrics

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mysql-exporter-secret
  namespace: prometheus  # 建议和Prometheus在同一个命名空间
type: Opaque
stringData:
  .my.cnf: |
    [client]
    user=exporter
    password=xxxxx
    host=xxx.xx.xxx.xxxx
    port=xxx

部署blackbox-exporter服务监控网站可用性

还需要在prometheus配置集群外的自动发现，即上面编辑的vm-scrape-secret.yaml文件

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  namespace: prometheus

spec:
  replicas: 1

  selector:
    matchLabels:
      app: blackbox-exporter

  template:
    metadata:
      labels:
        app: blackbox-exporter

    spec:
      containers:
      - name: blackbox-exporter

        image: harbor.ylbeijing.com/monitor/blackbox-exporter:v0.27.0

        args:
        - --config.file=/etc/blackbox/blackbox.yml

        ports:
        - containerPort: 9115
          name: metrics

        volumeMounts:
        - name: config
          mountPath: /etc/blackbox

      volumes:
      - name: config
        configMap:
          name: blackbox-config
      imagePullSecrets:
      - name: monitor-secret

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  namespace: prometheus

  labels:
    monitoring: enabled

spec:
  selector:
    app: blackbox-exporter

  ports:
  - name: metrics
    port: 9115
    targetPort: 9115

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-config
  namespace: prometheus

data:
  blackbox.yml: |
    modules:

      http_2xx:
        prober: http
        timeout: 5s
        http:
          preferred_ip_protocol: ip4

      tcp_connect:
        prober: tcp
        timeout: 5s

      icmp:
        prober: icmp
        timeout: 5s