【问题解决】Hadoop 执行 MapReduce卡死 问题

博客指出运行MapReduce时卡死的原因是Yarn可分配的内存不足。针对不同情况给出解决方案,若在虚拟机配置的Hadoop,可编辑yarn - site.xml文件并重启yarn;若在CDH,搜索后重启即可解决问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

运行MapReduce 卡死 , 截图如下

在这里插入图片描述

原因是 Yarn 可分配的内存不足

1.如果是在 虚拟机配置的 Hadoop

直接 编辑 yarn-site.xml 文件

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
     <value>2048</value>
</property>

注意 : 单位大小是 Mb , 2048 = 2G , 如果你的内存是 4G , 建议设置成 4G 或者 8G

重启 yarn即可

2.如果是在CDH

在这里插入图片描述
在这里插入图片描述
搜索 yarn.nodemanager.resource.memory-mb
在这里插入图片描述

然后重启即可

在这里插入图片描述

完美解决 !

[root@master exp3]# hadoop jar /export/servers/hadoop-3.4.0/share/hadoop/tools/lib/hadoop-streaming-3.4.0.jar -D mapreduce.job.reduces=1 -D stream.map.output.field.separator=\t -D stream.num.map .output.key.fields=1 -mapper "python /export/data/chap-5/exp3/Mapper.py" -reducer "python /export/data/chap-5/exp3/Reducer.py" -input /exp1/input -output /exp1/output_$(date +%s) -file ./Mapper.py -file ./Reducer.py 2025-05-31 16:24:06,054 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead. packageJobJar: [./Mapper.py, ./Reducer.py, /tmp/hadoop-unjar2796039736802025778/] [] /tmp/streamjob2856802959225773798.jar tmpDir=null 2025-05-31 16:24:06,522 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.24.67:8032 2025-05-31 16:24:06,621 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.24.67:8032 2025-05-31 16:24:06,844 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1748671547381_0012 2025-05-31 16:24:07,087 INFO mapred.FileInputFormat: Total input files to process : 1 2025-05-31 16:24:07,123 INFO mapreduce.JobSubmitter: number of splits:2 2025-05-31 16:24:07,218 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1748671547381_0012 2025-05-31 16:24:07,218 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-05-31 16:24:07,334 INFO conf.Configuration: resource-types.xml not found 2025-05-31 16:24:07,334 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-05-31 16:24:07,379 INFO impl.YarnClientImpl: Submitted application application_1748671547381_0012 2025-05-31 16:24:07,404 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1748671547381_0012/ 2025-05-31 16:24:07,405 INFO mapreduce.Job: Running job: job_1748671547381_0012 2025-05-31 16:24:13,507 INFO mapreduce.Job: Job job_1748671547381_0012 running in uber mode : false 2025-05-31 16:24:13,508 INFO mapreduce.Job: map 0% reduce 0% 2025-05-31 16:24:19,780 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:19,816 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:23,906 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:23,912 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:27,978 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:29,025 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:33,127 INFO mapreduce.Job: map 50% reduce 100% 2025-05-31 16:24:34,137 INFO mapreduce.Job: map 100% reduce 100% 2025-05-31 16:24:34,145 INFO mapreduce.Job: Job job_1748671547381_0012 failed with state FAILED due to: Task failed task_1748671547381_0012_m_000001 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-05-31 16:24:34,197 INFO mapreduce.Job: Counters: 14 Job Counters Failed map tasks=7 Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=8 Other local map tasks=6 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=23810 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=23810 Total vcore-milliseconds taken by all map tasks=23810 Total megabyte-milliseconds taken by all map tasks=24381440 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 2025-05-31 16:24:34,197 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!
06-01
评论 3
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

兀坐晴窗独饮茶

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值