【Hadoop】MapReduce Job Submission Files - 代码描述

最新推荐文章于 2019-12-19 16:34:29 发布

原创最新推荐文章于 2019-12-19 16:34:29 发布 · 1.2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hadoop

Hadoop 专栏收录该内容

19 篇文章

订阅专栏

本文详细介绍了 Hadoop 中 Job 提交的具体流程，包括复制 Job 的 JAR 文件到 JobTracker 的文件系统、生成输入切片、写入切片文件及配置文件等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Copy Job jar File

org.apache.hadoop.mapred.JobClient.submitJobInternal() --> copyAndConfigureFiles()

    String originalJarPath = job.getJar();

    //junz-20130713: Copy job jar file to JT's file system
    if (originalJarPath != null) {           // copy jar to JobTracker's fs
      // use jar name if job is not named. 
      if ("".equals(job.getJobName())){
        job.setJobName(new Path(originalJarPath).getName());
      }
      Path submitJarFile = JobSubmissionFiles.getJobJar(submitJobDir);
      job.setJar(submitJarFile.toString());
      fs.copyFromLocalFile(new Path(originalJarPath), submitJarFile);
      fs.setReplication(submitJarFile, replication);
      fs.setPermission(submitJarFile, 
          new FsPermission(JobSubmissionFiles.JOB_FILE_PERMISSION));
    }

Generate InputSplit

org.apache.hadoop.mapred.JobClient.submitJobInternal() --> writeSplits() --> writeNewSplits()

    InputFormat<?, ?> input =
      ReflectionUtils.newInstance(job.getInputFormatClass(), conf);

    List<InputSplit> splits = input.getSplits(job);
    T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);

    // sort the splits into order based on size, so that the biggest
    // go first
    Arrays.sort(array, new SplitComparator());

Write Split Files

JobClient.submitJobInternal() --> writeSplits() --> writeNewSplits() --> JobSplitWriter.createSplitFiles()

    FSDataOutputStream out = createFile(fs, 
        JobSubmissionFiles.getJobSplitFile(jobSubmitDir), conf);
    SplitMetaInfo[] info = writeNewSplits(conf, splits, out);
    out.close();
    writeJobSplitMetaInfo(fs,JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir), 
        new FsPermission(JobSubmissionFiles.JOB_FILE_PERMISSION), splitVersion,
        info);

Write Job Config File

JobClient.submitJobInternal()

          FSDataOutputStream out = 
            FileSystem.create(fs, submitJobFile,
                new FsPermission(JobSubmissionFiles.JOB_FILE_PERMISSION));

          try {
            jobCopy.writeXml(out);
          } finally {
            out.close();
          }