mapreduce job fail when submitted from windows machine

最新推荐文章于 2024-02-17 19:11:27 发布

原创最新推荐文章于 2024-02-17 19:11:27 发布 · 1.2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #mapreduce #cross-platform

java 同时被 2 个专栏收录

31 篇文章

订阅专栏

big data

10 篇文章

订阅专栏

本文解决了一个从Windows客户端提交MapReduce作业到Linux/Unix服务器时遇到的问题，通过设置配置项“mapreduce.app-submission.cross-platform”为true，成功避免了因作业提交平台不一致导致的异常。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

参照http://stackoverflow.com/questions/24075669/mapreduce-job-fail-when-submitted-from-windows-machine

遇到的问题如下：

2014-06-06 08:32:37,684 [main] INFO  Job.monitorAndPrintJob  - Job job_1399458460502_0053 running in uber mode : false
2014-06-06 08:32:37,704 [main] INFO  Job.monitorAndPrintJob  -  map 0% reduce 0%
2014-06-06 08:32:37,717 [main] INFO  Job.monitorAndPrintJob  - Job job_1399458460502_0053 failed with state FAILED due to: Application application_1399458460502_0053 failed 2 times due to AM Container for appattempt_1399458460502_0053_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
    at org.apache.hadoop.util.Shell.run(Shell.java:379)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)


.Failing this attempt.. Failing the application.

java.io.IOException: Job failed!

可以在java里设置

public Configuration config() {
Configuration conf = new YarnConfiguration();
conf.set("fs.defaultFS", HDFS);
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.address", “c0004650.itcs.hp.com:8032”);
conf.set("yarn.resourcemanager.scheduler.address", “c0004650.itcs.hp.com:8030”);
conf.set("mapreduce.app-submission.cross-platform", "true");
// put all of the third party jars like this, but these jars should be
// put on hadoop clusters.
// conf.set("mapreduce.application.classpath",
// "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,/opt/mount/learn/mahout-1.0-lib/*");
return conf;
}

也可以

(mapred-core.xml)

    <property>
      <description>If enabled, user can submit an application cross-platform
      i.e. submit an application from a Windows client to a Linux/Unix server or
      vice versa.
      </description>
        <name>mapreduce.app-submission.cross-platform</name>
        <value>false</value>
    </property>

By setting the value to true the error was gone