Hadoop3.2.0HA集群搭建后运行wordcount遇到错误

最新推荐文章于 2024-08-04 22:16:38 发布

吴杰JAVA

最新推荐文章于 2024-08-04 22:16:38 发布

阅读量1.6k

点赞数 3

CC 4.0 BY-SA版权

分类专栏： hadoop 文章标签： hadoop 大数据 java

本文链接：https://blog.youkuaiyun.com/u013783065/article/details/99699421

hadoop 专栏收录该内容

1 篇文章

订阅专栏

异常1：

Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory

解决方法：在中添加adoop-3.0.0/libexec/hadoop-config.sh文件中添加JAVA_HOME环境变量（export）

异常2：

Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
解决方法：按错误提示，在mapred-site.xml配置文件中添加hadoop根目录

异常3：

AM Container for appattempt_XXXXXX exited with exitCode: -103
Failing this attempt.Diagnostics

解决方法：在yarn-site.xml中添加以下配置
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>5</value>
<description>Ratio between virtual memory to physical memory when setting memory limits
for containers. Container allocations are expressed in terms of physical memory, and
virtual memory usage is allowed to exceed this allocation by this ratio.</description>
</property>

错误1：

ERROR: JAVA_HOME is not set and could not be found.

解决办法：

进入hadoop/libexec/编辑hadoop-config.sh添加java_home，最好重启一下即可！

错误2：

org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does
解决办法：

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

编写天气案例时遇到的一个揪心的bug：

Caused by: java.io.EOFException
   at java.io.DataInputStream.readInt(DataInputStream.java:392)
   at com.hafc.hadoop.weather.pojo.Weather.readFields(Weather.java:60)
   at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)
   ... 14 more
这个其实是我在在序列化和反序列化是类型不一致导致的。提醒自己以后一定要注意这个问题。

每个人遇到的错误都有可能不一致，如果对你有帮助，请点赞，如果你出现了这里没有的错误，可以留下评论，如果解决了海清一并留下解决方式，我们共同学习！