Yarn作业信息的获取

最新推荐文章于 2024-08-10 02:40:24 发布

asin929

最新推荐文章于 2024-08-10 02:40:24 发布

阅读量1w

点赞数 4

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签： Yarn 作业 job

本文链接：https://blog.youkuaiyun.com/u012948976/article/details/51623228

Hadoop 专栏收录该内容

15 篇文章

订阅专栏

本文介绍了三种获取Yarn作业信息的方式：1) 通过查看HDFS中"/user/history/done"目录下的作业记录文件，包含job.jhist和job.xml；2) 使用REST API，包括MapReduce History Server和ResourceManager等的API来查询作业详情；3) 利用Hadoop命令如`mapred job`和`yarn application`进行操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

方式1–查看作业记录文件

CDH中，在HDFS的”/user/history/done”目录下，包含了全部已完成的MR作业，”done_intermediate”包含了全部正在进行的作业。

在”/user/history/done”目录下，每个job有两个文件：job.jhist和job.xml，job.jhist是作业运行过程的详细记录，格式为json。job.xml是作业的配置文件，两者的示例结构如下。

job–.jhist文件结构

task信息（每行一个）

{"type":"TASK_STARTED","event":{"org.apache.hadoop.mapreduce.jobhistory.TaskStarted":{"taskid":"task_1465461051654_0001_r_000002","taskType":"REDUCE","startTime":1465461143745,"splitLocations":""}}}

counts（最后一行）

{"name":"DATA_LOCAL_MAPS","displayName":"Data-local map tasks","value":1},{"name":"SLOTS_MILLIS_MAPS","displayName":"Total time spent by all maps in occupied slots (ms)","value":2229},{"name":"SLOTS_MILLIS_REDUCES","displayName":"Total time spent by all reduces in occupied slots (ms)","value":17791},{"name":"MILLIS_MAPS","displayName":"Total time spent by all map tasks (ms)","value":2229},{"name":"MILLIS_REDUCES","displayName":"Total time spent by all reduce tasks (ms)","value":17791},{"name":"VCORES_MILLIS_MAPS","displayName":"Total vcore-seconds taken by all map tasks","value":2229},{"name":"VCORES_MILLIS_REDUCES","displayName":"Total vcore-seconds taken by all reduce tasks","value":17791},{"name":"MB_MILLIS_MAPS","displayName":"Total megabyte-seconds taken by all map tasks","value":2282496},{"name":"MB_MILLIS_REDUCES","displayName":"Total megabyte-seconds taken by all reduce tasks","value":18217984}]},

job–.xml文件结构
- 输入
  - name: mapreduce.input.fileinputformat.inputdir
  - value:/user/hdfs/test
- 输出
  - name:mapreduce.output.fileoutputformat.outputdir
  - value:/user/hdfs/out3

方式2–REST API

Hadoop YARN - Introduction to the web services REST API’s通过一个示例作业的完整运行过程，介绍了其所用到的全部REST API，包括

以查看某一作业详情为例，

request

curl --compressed -H "Accept: application/json" -X GET "http://lyhadoop4.com:8088/ws/v1/cluster/apps/application_1465461051654_0001"

response

{
  "app": {
    "id": "application_1465461051654_0001",
    "user": "hdfs",
    "name": "word count",
    "queue": "root.hdfs",
    "state": "FINISHED",
    "finalStatus": "SUCCEEDED",
    "progress": 100,
    "trackingUI": "History",
    "trackingUrl": "http://lyhadoop4.com:8088/proxy/application_1465461051654_0001/jobhistory/job/job_1465461051654_0001",
    "diagnostics": "",
    "clusterId": 1465461051654,
    "applicationType": "MAPREDUCE",
    "applicationTags": "",
    "startedTime": 1465461140788,
    "finishedTime": 1465461160863,
    "elapsedTime": 20075,
    "amContainerLogs": "http://lyhadoop6.com:8042/node/containerlogs/container_1465461051654_0001_01_000001/hdfs",
    "amHostHttpAddress": "lyhadoop6.com:8042",
    "allocatedMB": -1,
    "allocatedVCores": -1,
    "runningContainers": -1,
    "memorySeconds": 59076,
    "vcoreSeconds": 53,
    "preemptedResourceMB": 0,
    "preemptedResourceVCores": 0,
    "numNonAMContainerPreempted": 0,
    "numAMContainerPreempted": 0
  }
}