Hadoop源码分析33 Child的主要流程

最新推荐文章于 2023-10-15 20:25:06 发布

荒岛码农

最新推荐文章于 2023-10-15 20:25:06 发布

阅读量743

点赞数

CC 4.0 BY-SA版权

分类专栏：大数据-云计算

本文链接：https://blog.youkuaiyun.com/lihui6636/article/details/27303789

大数据-云计算专栏收录该内容

133 篇文章

订阅专栏

本文详细介绍了如何在Hadoop中配置调试参数，包括使用-Xmx200m、-Xdebug、-Xrunjdwp等选项，并通过提交WordCount作业来生成两个Map和两个Reduce任务。此外，文章还涉及了执行Setup任务、生成JobToken文件、设置工作目录、启动TaskReporter线程以及处理任务状态更新和ping操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

添加调试参数：

＜property＞

＜name＞mapred.child.java.opts＜/name＞

＜value＞-Xmx200m-Xdebug-Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=y＜/value＞

＜/property＞

提交作业:

hadoop jar /opt/hadoop-1.0.0/hadoop-examples-1.0.0.jar wordcount /user/admin/in/yellow2.txt /user/admin/out/128

生成2个Map、2个Reduce任务。

执行Setup任务：

args= [127.0.0.1,40996, attempt_201404282305_0001_m_000003_0,/opt/hadoop-1.0.0/logs/userlogs/job_201404282305_0001/attempt_201404282305_0001_m_000003_0,-1093852866]

变量:

jvmId=JVMId{id=-1093852866,isMap=true,jobId=job_201404282305_0001}

cwd=/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/attempt_201404282305_0001_m_000003_0/work

jobTokenFile=/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/jobToken

taskOwner=job_201404282305_0001

umbilical=(TaskUmbilicalProtocol)RPC.getProxy(TaskUmbilicalProtocol.class,

TaskUmbilicalProtocol.versionID,address,defaultConf);

context=JvmContext{jvmId=jvm_201404282305_0001_m_-1093852866,pid="28737"}

myTask =JvmTask {shouldDie=false,

t=MapTask { taskId=attempt_201404282305_0001_m_000003_0,jobCleanup=false, jobSetup=true, taskCleanup=false, jobFile="/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/job.xml"}, taskStatus=MapTaskStatus{runState=UNASSIGNED}}

job= JobConf{Configuration:core-default.xml, core-site.xml, mapred-default.xml,mapred-site.xml,/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/job.xml}

currentJobSegmented= false

isCleanup =false

DistributedFileSystem的 workingDir=hdfs://server1:9000/user/admin

启动一个TaskReporter线程，检查Task.progressFlag变量(AtomicBoolean)，true则通过RPC汇报statusUpdate(taskId,taskStatus,jvmContext)，false则通过RPC进行ping(askId,jvmContext).

Task的jobContext={conf={Configuration:core-default.xml, core-site.xml, mapred-default.xml,mapred-site.xml, hdfs-default.xml, hdfs-site.xml,/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/job.xml}，

job=JobConf{Configuration:core-default.xml, core-site.xml, mapred-default.xml,mapred-site.xml, hdfs-default.xml, hdfs-site.xml,/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/job.xml},

jobId={job_201404282305_0001}}

Task的taskContext={conf={Configuration:core-default.xml, core-site.xml, mapred-default.xml,mapred-site.xml, hdfs-default.xml, hdfs-site.xml,/tmp/hadoop-admin/mapred/local/taskTracker/admin/jobcache/job_201404282305_0001/job.xml}，taskId=attempt_201404282305_0001_m_000003_0，jobId=job_201404282305_0001，status=""}

outputFormat=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat@7099c91f

committer={outputFileSystem=DFS[DFSClient[clientName=DFSClient_attempt_201404282305_0001_m_000003_0,ugi=admin]]，outputpath=/user/admin/out/128, workPath=hdfs://server1:9000/user/admin/out/128/_temporary/_attempt_201404282305_0001_m_000003_0}

Task的resourceCalculator=org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2288e718

Task的initCpuCumulativeTime=13620

建立文件夹 /user/admin/out/128/_temporary后则完成.