Hadoop中MapReduce之Job提交和切片信息详解

最新推荐文章于 2021-08-13 13:49:49 发布

Hao_JunJie

最新推荐文章于 2021-08-13 13:49:49 发布

阅读量1.7k

点赞数 7

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签： hadoop

本文链接：https://blog.youkuaiyun.com/Hao_JunJie/article/details/115212620

下图是wordCount驱动类，从源码方式看它是如何进行提交的

进入waitForCompletion 方法之后

当state为DEFINE 进行submit() 进行提交

ensureState(JobState.DEFINE):确保job的状态为DEFINE

setUSerNewAPI(); 使用新的API

connect() 建立连接：是提交到YARN集群还是Local 如下图：

进入connect() 方法

返回Cluster对象：return new Cluster(getConfiguration())

Cluster 调用initialize(jobTrackAddr,conf)

调用 initProviderList(); 初始化提交者

providerList 值为：YarnClientProtocolProvider 和 LocalClientProtocolProvider

继续图六：

进入create(conf)方法（是个抽象方法)

Yarn的实现:

本地的实现:

如果连接的是Yarn集群 clientProtocol就是 YARNRunner 如果连接的是local clientProtocol就是LocalJobRunner

connect() 方法结束后

进行提交 submitJobInternal（Job.this,cluster）：如下代码：截图放不下，改成代码段

JobStatus submitJobInternal(Job job, Cluster cluster) 
  throws ClassNotFoundException, InterruptedException, IOException {

    //validate the jobs output specs 
    checkSpecs(job);

    Configuration conf = job.getConfiguration();
    addMRFrameworkToDistributedCache(conf);

    Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
    //configure the command line options correctly on the submitting dfs
    InetAddress ip = InetAddress.getLocalHost();
    if (ip != null) {
      submitHostAddress = ip.getHostAddress();
      submitHostName = ip.getHostName();
      conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
      conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
    }
    JobID jobId = submitClient.getNewJobID();
    job.setJobID(jobId);
    Path submitJobDir = new Path(jobStagingArea, jobId.toString());
    JobStatus status = null;
    try {
      conf.set(MRJobConfig.USER_NAME,
          UserGroupInformation.getCurrentUser().getShortUserName());
      conf.set("hadoop.http.filter.initializers", 
          "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer");
      conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, submitJobDir.toString());
      LOG.debug("Configuring job " + jobId + " with " + submitJobDir 
          + " as the submit dir");
      // get delegation token for the dir
      TokenCache.obtainTokensForNamenodes(job.getCredentials(),
          new Path[] { submitJobDir }, conf);
      
      populateTokenCache(conf, job.getCredentials());