[JMX]jmx监控hadoop3.x和Hadoop2.x（一步到位）

背景

Prometheus+jmx_prometheus_javaagent-0.16.1.jar+hadoop3.2.1

jmx监控Hadoop3.x

1 环境的准备

jmx下载地址：https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

Prometheus下载地址：
https://prometheus.io/download/

hadoop下载地址：
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
安装过程就不一一赘述了，本文着重jmx使用

2 jmx在hadoop中的编写（或见 一步到位）

2.1 hdfs——jmx

[root@bdp01 bin]# pwd
/opt/hadoop/bin
[root@bdp01 bin]# ll
total 1040
-rwxrwxrwx. 1 1001 mysql 442480 Sep 11  2019 container-executor
-rwxrwxrwx. 1 1001 mysql   8707 Sep 10  2019 hadoop
-rwxrwxrwx. 1 1001 mysql  11265 Sep 10  2019 hadoop.cmd
-rwxrwxrwx. 1 1001 mysql  12573 May  7 14:36 hdfs
-rwxrwxrwx. 1 1001 mysql   8081 Sep 11  2019 hdfs.cmd
-rw-r--r--. 1 root root       0 May  5 17:45 kill
-rwxrwxrwx. 1 1001 mysql   6240 May  7 10:59 mapred
-rwxrwxrwx. 1 1001 mysql   6311 Sep 11  2019 mapred.cmd
-rw-------. 1 root root       0 May  5 17:45 nohup.out
-rwxrwxrwx. 1 1001 mysql  29184 Sep 11  2019 oom-listener
-rwxrwxrwx. 1 1001 mysql 484264 Sep 11  2019 test-container-executor
-rwxrwxrwx. 1 1001 mysql  13157 May  7 15:34 yarn
-rwxrwxrwx. 1 1001 mysql  12840 Sep 11  2019 yarn.cmd

vi hdfs

如图

## hdfs  monitor
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9221 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5221:/opt/hadoop/nn.yaml $HADOOP_NAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9222 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5222:/opt/hadoop/dn.yaml $HADOOP_DATANODE_OPTS"

export HADOOP_JOURNALNODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9225 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5225:/opt/hadoop/journalnode.yaml $HADOOP_JOURNALNODE_OPTS"

export HADOOP_ZKFC_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9224 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5224:/opt/hadoop/zkfc.yaml $HADOOP_ZKFC_OPTS"

2.2 yarn——jmx

[root@bdp01 bin]# ll
total 1040
-rwxrwxrwx. 1 1001 mysql 442480 Sep 11  2019 container-executor
-rwxrwxrwx. 1 1001 mysql   8707 Sep 10  2019 hadoop
-rwxrwxrwx. 1 1001 mysql  11265 Sep 10  2019 hadoop.cmd
-rwxrwxrwx. 1 1001 mysql  12573 May  7 14:36 hdfs
-rwxrwxrwx. 1 1001 mysql   8081 Sep 11  2019 hdfs.cmd
-rw-r--r--. 1 root root       0 May  5 17:45 kill
-rwxrwxrwx. 1 1001 mysql   6240 May  7 10:59 mapred
-rwxrwxrwx. 1 1001 mysql   6311 Sep 11  2019 mapred.cmd
-rw-------. 1 root root       0 May  5 17:45 nohup.out
-rwxrwxrwx. 1 1001 mysql  29184 Sep 11  2019 oom-listener
-rwxrwxrwx. 1 1001 mysql 484264 Sep 11  2019 test-container-executor
-rwxrwxrwx. 1 1001 mysql  13157 May  7 15:34 yarn
-rwxrwxrwx. 1 1001 mysql  12840 Sep 11  2019 yarn.cmd
[root@bdp01 bin]# pwd
/opt/hadoop/bin

如图

vi yarn

export YARN_RESOURCEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6464 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=17104:/opt/hadoop/rm.yaml $YARN_RESOURCEMANAGER_OPTS"

export YARN_NODEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6474 -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=17105:/opt/hadoop/nm.yaml $YARN_NODEMANAGER_OPTS"

3.jmx相关参数的yaml文件编写

如图把jar包放到hdfs和yarn配置的相同路径位置，并编写yaml文件。

举两个例子，其实只要端口号修改一下，跟上面hdfs和yarn端口对应就行（也可以看后面的最简yaml编写）

vi nn.yaml

startDelaySeconds: 0
hostPort: 192.168.x.xxx:9221
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

 vi dn.yaml

startDelaySeconds: 0
hostPort: 192.168.0.101:9222
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

4.Prometheus部分

 vi prometheus.yml

这样写的好处是今后可以动态修改Prometheus配置，无需重启Prometheus

 - job_name: 'hdfs'
    scrape_interval: 80s
    scrape_timeout: 60s
    file_sd_configs:
     - files:
       - /opt/prometheus/prom/hdfs.json
       refresh_interval: 5s

上文中有个json文件，我编写了一个json，举个hdfs的例子。hdfs包括namenode，datanode，journalnode和zkfc

vi hdfs.json

[
  {
      "targets": [
         "192.168.x.xxx:5221"
      ],
      "labels": {
         "project_name": "namenode"
      }
   },
 {
      "targets": [
         "192.168.x.xxx:5222"
      ],
      "labels": {
         "project_name": "datanode"
      }
   },
   {
      "targets": [
         "192.168.x.xxx:5225"
      ],
      "labels": {
         "project_name": "historyserver"
      }
   },
   {
      "targets": [
         "192.168.x.xxx:5224"
      ],
      "labels": {
         "project_name": "zkfc"
      }
   }
]

5.最后操作启停

hdfs yarn Prometheus 重启
查看Prometheus

如果安装成功，http://ip:port也会有数据，如图

一步到位：Hadoop3.x的jmx配置在hadoop-env.sh，直接重启角色

export HADOOP_NAMENODE_OPTS=" -javaagent:/opt/hadoop/jmx_prometheus_javaagent-0.16.1.jar=5229:/opt/hadoop/nn.yaml $HADOOP_NAMENODE_OPTS"

一步到位：jmx监控hadoop2.x

vi etc/hadoop/hadoop-env.sh

export HADOOP_IDENT_STRING=$USER
export HADOOP_JOURNALNODE_OPTS="-javaagent:/opt/bdp/monitor/jmx_prometheus_javaagent-0.16.1.jar=2101:/opt/bdp/monitor/hadoop/journalnode.yaml $HDFS_JOURNALNODE_OPTS"
export HADOOP_NAMENODE_OPTS="-javaagent:/opt/bdp/monitor/jmx_prometheus_javaagent-0.16.1.jar=2012:/opt/bdp/monitor/hadoop/namenode.yaml $HDFS_NAMENODE_OPTS"
export JAVA_HOME=/usr/local/java
export HADOOP_DATANODE_OPTS=" -javaagent:/opt/bdp/monitor/jmx_prometheus_javaagent-0.16.1.jar=2014:/opt/bdp/monitor/hadoop/datanode.yaml $HDFS_DATANODE_OPTS"

然后启动对应的角色，用浏览器访问 http://ip:port就可以 ，如http:192.168.80.161:2101,
或者shell命令： curl http:192.168.80.161:2101

最简yanl文件编写

栗子：

vi /opt/bdp/monitor/hadoop/datanode.yaml

rules:
    - pattern: '.*'

vi /opt/bdp/monitor/hadoop/namenode.yaml

rules:
    - pattern: '.*'

此时的jmx监控如下：只有一个端口号需要传入

export HADOOP_NAMENODE_OPTS="-javaagent:/opt/bdp/monitor/jmx_prometheus_javaagent-0.16.1.jar=2012:/opt/bdp/monitor/hadoop/namenode.yaml $HDFS_NAMENODE_OPTS"

其他

bash -x  hdfs --daemon stop datanode