[Hadoop] “Too many fetch-failures” or “reducer stucks” issue

最新推荐文章于 2021-10-11 17:48:15 发布

转载最新推荐文章于 2021-10-11 17:48:15 发布 · 1.6k 阅读

文章标签：

#hadoop #network #parameters #jobs #binding #linux

Hadoop学习专栏收录该内容

19 篇文章

订阅专栏

本文提供了解决Hadoop集群中Reducer任务挂起的问题解决方案，涉及检查Linux网络配置、Hadoop配置文件、主机名绑定及节点间通信等关键步骤。详细指导帮助您诊断并解决该常见问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

转自：http://lykke.iteye.com/blog/1405838

I post the solution here to help any ‘Hadoopers’ that have the same problem. This issue had been asked a lot on Hadoop mailing list but no answer was given so far.
After installing Hadoop cluster and trying to run some jobs, you may see the Reducers stuck and TaskTracker log on one of the Worker node shows these messages :

   Java代码  
   
  
 INFO org.apache.hadoop.mapred.TaskTracker: task_200801281756_0001_r_000000_0 0.2727273% reduce > copy (9 of  
 11 at 0.00 MB/s) >  
 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.2727273% reduce > copy (9 of  
 11 at 0.00 MB/s) >  
 INFO org.apache.hadoop.mapred.TaskTracker: task_0001_r_000000_0 0.2727273% reduce > copy (9 of  
 11 at 0.00 MB/s) >  
 INFO org.apache.hadoop.mapred.JobInProgress: Too many fetch-failures for output of task: task_001_r_000000_0 … killing it  

The Reducer was failed to copy data through the HDFS, what we should do is to double check your Linux network and Hadoop configuration :

1. Make sure that all the needed parameters are configured in hadoop-site.xml, and all the worker nodes should have the same content of this file.
2. URI for TaskTracker and HDFS should use hostname instead of IP address. I saw some instances of Hadoop cluster using IP address for the URI, they can start all the services and execute the jobs, but the task never finished successfully.
3. Check the file /etc/hosts on all the nodes and make sure that you’re binding the host name to its network IP, not the local one (127.0.0.1), don’t forget to check that all the nodes are able to communicate to the others using their hostname.
Anyway, it doesn’t make sense to me when Hadoop always try to resolve an IP address using the hostname. I consider this is a bug of Hadoop and hope they will solve it in next stable version.