hadoop异常 error in shuffle in fetcher

最新推荐文章于 2023-11-12 23:33:28 发布

原创最新推荐文章于 2023-11-12 23:33:28 发布 · 2k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #异常 #hive

hive 专栏收录该内容

5 篇文章

订阅专栏

本文介绍了解决Hadoop MapReduce任务中reduce shuffle阶段出现的内存溢出错误的方法。通过调整参数限制shuffle过程中的内存使用，避免了因数据量过大导致的异常。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

错误信息:Error:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5
解决方法：限制reduce的shuffle内存使用
hive：set mapreduce.reduce.shuffle.memory.limit.percent=0.1;
MR：job.getConfiguration().setStrings(“mapreduce.reduce.shuffle.memory.limit.percent”, “0.1”);
原理分析：reduce会在map执行到一定比例启动多个fetch线程去拉取map的输出结果，放到reduce的内存、磁盘中，然后进行merge。当数据量大时，拉取到内存的数据就会引起OOM，所以此时要减少fetch占内存的百分比，将fetch的数据直接放在磁盘上。
mapreduce.reduce.shuffle.memory.limit.percent：每个fetch取到的map输出的大小能够占的内存比的大小。默认是0.25。因此实际每个fetcher的输出能放在内存的大小是reducer的java heap size*0.9*0.25。

----

Task ID:

  task_1465962505870_540836_r_000713



URL:

  http://bigdata-arch-nm00.bh:18088/taskdetails.jsp?jobid=job_1465962505870_540836&tipid=task_1465962505870_540836_r_000713

-----

Diagnostic Messages for this Task:

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#5

    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)

    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)

    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:415)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

Caused by: java.lang.OutOfMemoryError: Java heap space

    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)

    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)

    at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)

    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)

    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)

    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)



Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143





FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched: