JMX Port Monitoring for Cloudera CDH4

最新推荐文章于 2020-09-02 15:15:27 发布

puffsun

最新推荐文章于 2020-09-02 15:15:27 发布

阅读量146

点赞数

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签：大数据 json jira

本文链接：https://blog.youkuaiyun.com/puffsun/article/details/84456694

Hadoop 专栏收录该内容

57 篇文章

订阅专栏

本文介绍如何通过配置 Hadoop 的 JMX 选项实现远程监控 NameNode 的状态，并提供了具体的命令示例。此外，还介绍了如何查询 NameNode 的详细信息，包括线程数量、磁盘使用情况等。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

By default, JMX info is only accessible via the JMX JSON servet (https://issues.apache.org/jira/browse/HDFS-2083). For example, you can query the NameNode status via:

curl -i http://n1.example.com:50070/jmx

or be more specifically:

curl -i http://n1.example.com:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo

or use your browser as an alternative. The output of the last query is:

HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Mon, 08 Jul 2013 14:56:16 GMT
Date: Mon, 08 Jul 2013 14:56:16 GMT
Pragma: no-cache
Expires: Mon, 08 Jul 2013 14:56:16 GMT
Date: Mon, 08 Jul 2013 14:56:16 GMT
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.2)

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "Threads" : 65,
    "Total" : 78674993973,
    "ClusterId" : "cluster50",
    "BlockPoolId" : "BP-1455653694-192.168.1.201-1373100840050",
    "Used" : 219303936,
    "PercentUsed" : 0.2787467,
    "PercentRemaining" : 84.45627,
    "Version" : "2.0.0-cdh4.3.0, r48a9315b342ca16de92fcc5be95ae3650629155a",
    "Free" : 66445963264,
    "Safemode" : "",
    "UpgradeFinalized" : true,
    "NonDfsUsedSpace" : 12009726773,
    "BlockPoolUsedSpace" : 219303936,
    "PercentBlockPoolUsed" : 0.2787467,
    "TotalBlocks" : 228,
    "TotalFiles" : 385,
    "NumberOfMissingBlocks" : 0,
    "LiveNodes" : "{\"n3.example.com\":{\"numBlocks\":228,\"usedSpace\":73142272,\"lastContact\":2,\"capacity\":26224997991,\"nonDfsUsedSpace\":3542431335,\"adminState\":\"In Service\"},\"n2.example.com\":{\"numBlocks\":223,\"usedSpace\":73080832,\"lastContact\":1,\"capacity\":26224997991,\"nonDfsUsedSpace\":3546506855,\"adminState\":\"In Service\"},\"n1.example.com\":{\"numBlocks\":223,\"usedSpace\":73080832,\"lastContact\":0,\"capacity\":26224997991,\"nonDfsUsedSpace\":4920788583,\"adminState\":\"In Service\"}}",
    "DeadNodes" : "{}",
    "DecomNodes" : "{}",
    "NameDirStatuses" : "{\"failed\":{},\"active\":{\"/dfs/nn\":\"IMAGE_AND_EDITS\"}}"
  } ]

You can also configure hadoop-env.sh (usually located in /etc/hadoop/..., for example, the mapreduce1 configure file path is /etc/hadoop/conf.cloudera.hdfs1) in CDH4, update HADOOP_NAMENODE_OPTS with

-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=$HADOOP_CONF_DIR/jmxremote.password
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8004

Then connect to the JMX server at port 8004 with JConsole, VisualVM or use the built-in JMXGet tool as well.

[root@n1 conf.cloudera.hdfs1]# hadoop org.apache.hadoop.hdfs.tools.JMXGet --help
usage: jmxget options are:
 -help                           print help
 -localVM <VM's connector url>   connect to the VM on the same machine;
                                 use:
                                 jstat -J-Djstat.showUnsupported=true
                                 -snap <vmpid> | grep
                                 sun.management.JMXConnectorServer.address
                                 to find the url
 -port <mbean server port>       specify mbean server port, if missing -
                                 it will try to connect to MBean Server in
                                 the same VM
 -server <mbean server>          specify mbean server (localhost by
                                 default)
 -service <NameNode|DataNode>    specify jmx service (NameNode by default)

Above is the configuration of NameNode, you can configure the JMX monitoring to SecondaryNameNode, DataNode, JobTracker, TaskTracker just like the NameNode. See this blog post.