Prometheus+node_exporter+grafana实现监控数据可视化

安装
https://blog.youkuaiyun.com/qq_30038621/article/details/118293886
调适
https://www.cnblogs.com/fsckzy/p/12614176.html

prometheus.yml

HTTP 配置

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # 模块对应 blackbox.yml 
    static_configs:
      - targets:
        - http://baidu.com    # http
        - https://baidu.com   # https
        - http://xx.com:8080 # 8080端口的域名
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox安装在哪台机器

TCP 配置

  - job_name: blackbox_tcp
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - 192.168.1.2:280
        - 192.168.1.2:7013

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.99:9115 # Blackbox exporter.

告警规则

以下两条二选一

groups:
  - name: http
    rules:
    - alert: xxx域名解析失败
      expr: probe_success{instance="https://xx.com"} == 0
      for: 1m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失败"
    - alert: xxx域名解析失败
      expr: probe_http_status_code{instance="https://xx.com"} != 200
      for: 5m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失败"

自定义模块
有时可能对于某些 URL 需要带参数，如 header、body 之类的，就需要自定义一个模块，官方例子。

编辑 blackbox.yml

  http_2xx_wxjj:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Cookie: JSESSIONID=C123455dfdf
        sid: 41c912344555-24rkjkffd
        appid: 1221kj2h1k3hjk13hk
      body: '{}'

编辑 Prometheus.yml

  - job_name: 'blackbox_wxjl'
    metrics_path: /probe
    params:
      module: [http_2xx_wxjj]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList&user=%E9%BB%84%E5%AE%15
   # Target to probe with http.

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.18.11.154:9115  # The blackbox exporter's real hostname:port.

FAQ
需要安装多少个 blackbox_exporter ？
理论上只安装一个即可，在特别的网络环境，比如政务云，和第三方对接的时候，只有几台机器开通了网络，那就需要在那几台机器中的其中一台部署 blackbox_exporter，同时 Prometheus.yml 里的 replacement 填上相应的 ip