执行wordcount例子报异常

最新推荐文章于 2023-03-22 16:58:21 发布

原创最新推荐文章于 2023-03-22 16:58:21 发布 · 7.7k 阅读

11 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #异常

大数据专栏收录该内容

2 篇文章

订阅专栏

本文介绍了三种Hadoop任务失败的情况，并提供了相应的解决方法。包括设置JAVA_HOME环境变量、配置mapred-site.xml文件中的HADOOP_MAPRED_HOME路径以及调整yarn-site.xml中的虚拟内存限制。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

异常一：
2018-01-09 02:50:43,320 INFO mapreduce.Job: Job job_1515466125577_0002 failed with state FAILED due to: Application application_1515466125577_0002 failed 2 times due to AM Container for appattempt_1515466125577_0002_000002 exited with exitCode: 127
Failing this attempt.Diagnostics: [2018-01-09 02:50:42.706]Exception from container-launch.
Container id: container_1515466125577_0002_02_000001
Exit code: 127

[2018-01-09 02:50:42.750]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory

解决方法：在中添加adoop-3.0.0/libexec/hadoop-config.sh文件中添加JAVA_HOME环境变量

异常二：
2018-01-09 03:25:37,250 INFO mapreduce.Job: Job job_1515468264727_0001 failed with state FAILED due to: Application application_1515468264727_0001 failed 2 times due to AM Container for appattempt_1515468264727_0001_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2018-01-09 03:25:36.447]Exception from container-launch.
Container id: container_1515468264727_0001_02_000001
Exit code: 1

[2018-01-09 03:25:36.499]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
解决方法：按错误提示，在mapred-site.xml配置文件中添加hadoop根目录

异常三：
2018-01-09 03:55:18,863 INFO mapreduce.Job: Job job_1515470038765_0001 failed with state FAILED due to: Application application_1515470038765_0001 failed 2 times due to AM Container for appattempt_1515470038765_0001_000002 exited with exitCode: -103
Failing this attempt.Diagnostics: [2018-01-09 03:55:17.480]Container [pid=728,containerID=container_1515470038765_0001_02_000001] is running beyond virtual memory limits. Current usage: 146.5 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1515470038765_0001_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 741 728 728 728 (java) 634 19 2852925440 37258 /usr/local/jdk1.8.0_131/bin/java -Djava.io.tmpdir=/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1515470038765_0001/container_1515470038765_0001_02_000001/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/hadoop-3.0.0/logs/userlogs/application_1515470038765_0001/container_1515470038765_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
解决方法：在yarn-site.xml中添加以下配置
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>5</value>
<description>Ratio between virtual memory to physical memory when setting memory limits
for containers. Container allocations are expressed in terms of physical memory, and
virtual memory usage is allowed to exceed this allocation by this ratio.</description>
</property>