MapReduce job Shuffle 过程的ERROR

最新推荐文章于 2023-05-07 22:26:33 发布

ChaosJ

最新推荐文章于 2023-05-07 22:26:33 发布

阅读量1.5k

点赞数

CC 4.0 BY-SA版权

分类专栏： elasticsearch hadoop 文章标签： hadoop shuffle mapreduce error

本文链接：https://blog.youkuaiyun.com/ChaosJ/article/details/51146015

hadoop 同时被 2 个专栏收录

7 篇文章

订阅专栏

elasticsearch

3 篇文章

订阅专栏

本文详细介绍了如何通过调整MapReduce的shuffle输入缓冲区百分比参数来解决Job Shuffle阶段出现的内存溢出问题。通过将参数值从默认的0.7修改为0.6，并解释了相关参数的作用及其对优化内存使用的重要性。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.错误描述

error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#43
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1550)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:414)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:343)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:166)

2.问题解决

从error的stack信息可以看出是mapreduce 的job shuffle过程中发生的OOM，map的输出数据在reduce节点load的过多导致的OOM，所以将reduce的shuffle.input 由默认的0.7 改为0.6，该参数的变化需要根据实际情况确定。

  conf.set("mapreduce.reduce.shuffle.input.buffer.percent", "0.6");

3.相关参数

mapreduce.reduce.shuffle.input.buffer.percent ：

the percentage of the reducer's heap memory to be allocated for the circular buffer to store the intermedite outputs copied from multiple mappers.

mapreduce.reduce.shuffle.memory.limit.percent：

the maximum percentage of the above memory buffer that a single shuffle (output copied from single Map task) should take. The shuffle's size above this size will not be copied to the memory buffer, instead they will be directly written to the disk of the reducer.

mapreduce.reduce.shuffle.merge.percent：

the threshold percentage by where the in-memory merger thread will run to merge the available shuffle contents on the memory buffer into a single file and immediately spills the merged file into the disk.