Flume 监控部署

FLume监控部署— Flume_exporter

---

前言

Flume提供了四种监控方式：JMX Reporting、Ganglia Reporting、JSON Reporting、Custom Reporting.选择最简单的JSON Reporting方式，监控方案是
JSON Reporting + flume_exporter + Prometheus + Grafana。

一、JSON Reporting 设置

$ bin/flume-ng 代理 --conf-file example.conf --name a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

具体可查看官网文档

http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#monitoring

二、flume_exporter 安装

1.flume_exporter下载

下载地址 https://github.com/woozhijun/flume_exporter

2.编译

make build

./flume_exporter 

该处需要安装go环境进行编译 。

2.1 安装go 环境

go下载地址

解压配置环境变量

cd  /usr/local
tar -zxvf  go1.17.10.linux-amd64.tar.gz 
vim /etc/profile
# 在最后一行添加
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin
# 保存退出后source一下（vim 的使用方法可以自己搜索一下）
source /etc/profile

执行 go version 查看是否安装成功

2.2、make build

编译安装过程出现网络异常情况

cd flume_exporter-master/
make build 
go: downloading github.com/prometheus/promu v0.13.0
go get: github.com/prometheus/promu@v0.13.0: verifying module: github.com/prometheus/promu@v0.13.0: Get "https://sum.golang.org/lookup/github.com/prometheus/promu@v0.13.0": dial tcp 142.251.42.241:443: i/o timeout
make: *** [promu] 错误 1

修改使用国内镜像

go env -w GO111MODULE=on
go env -w GOPROXY=https://goproxy.cn,direct

安装依赖包

go  get github.com/Sirupsen/logrus
go: downloading github.com/Sirupsen/logrus v1.8.1
go get: github.com/Sirupsen/logrus@v1.8.1: parsing go.mod:
        module declares its path as: github.com/sirupsen/logrus
                but was required as: github.com/Sirupsen/logrus

下载其他依赖

go get github.com/prometheus/promu@v0.13.0 &&go get github.com/google/go-cmp@v0.5.2
go get github.com/prometheus/client_golang/prometheus &&go get github.com/prometheus/client_golang/prometheus/promhttp && go get github.com/prometheus/common/version && go get github.com/woozhijun/flume_exporter/exporter && go get gopkg.in/alecthomas/kingpin.v2

最后编译成功

make build  
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading github.com/cespare/xxhash v1.1.0
go get: installing executables with 'go get' in module mode is deprecated.
        To adjust and download dependencies of the current module, use 'go get -d'.
        To install using requirements of the current module, use 'go install'.
        To install ignoring the current module, use 'go install' with a version,
        like 'go install example.com/cmd@latest'.
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.
go get: upgraded github.com/alecthomas/units v0.0.0-20201120081800-1786d5ef83d4 => v0.0.0-20211218093645-b94a6e3cc137
go get: upgraded github.com/google/go-cmp v0.5.2 => v0.5.5
go get: upgraded github.com/google/go-querystring v1.0.0 => v1.1.0
go get: upgraded github.com/prometheus/client_golang v1.9.0 => v1.12.2
go get: upgraded github.com/prometheus/common v0.26.0 => v0.34.0
go get: upgraded github.com/prometheus/procfs v0.5.0 => v0.7.3
go get: upgraded github.com/prometheus/promu v0.12.0 => v0.13.0
go get: upgraded golang.org/x/net v0.0.0-20220225172249-27dd8689420f => v0.0.0-20220517181318-183a9ca12b87
go get: upgraded golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b => v0.0.0-20220411215720-9780585627b5
go get: upgraded golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 => v0.0.0-20220517195934-5e4e11fc645e
go get: upgraded google.golang.org/protobuf v1.25.0 => v1.28.0
>> building binaries
 >   flume_exporter-master

3、启动flume_export

Configuration: config.yml

样例参考
agents:
- name: "flume-agents"
  enabled: true
# multiple urls can be separated by ,  
  urls: ["http://localhost:34545/metrics"]   

启动服务

nohup ./flume_exporter-master --metric-file=./metrics.yml --config-file=./config.yml &

4、验证

curl方式查看是否存在数据
curl  -get http://localhost:9360/metrics

三、添加进Prometheus

prometheus.yml 添加配置信息

  - job_name: 'flume-discovery'
    static_configs:
      - targets:
        - xxx.xxx.xx.xx:9360
    metrics_path: /metrics

重启 prometheus
若开启了热更新配置则直接命令更新

热更新参数 
--web.enable-lifecycle

curl -X POST http://localhost:9090/-/reload

四、Grafana 展示

QUERY  FLUME_CHANNEL_ChannelSize{host="$host"}

参考文章：
https://blog.youkuaiyun.com/qq_21133131/article/details/117568214
https://blog.youkuaiyun.com/qq_27710549/article/details/100745914