关于INFO mapreduce.Job: Job卡死的解决办法

最新推荐文章于 2023-02-07 22:28:59 发布
原创 最新推荐文章于 2023-02-07 22:28:59 发布 · 5k 阅读 · 5 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

网上试了很多方法，无论怎么修改yarn-size.xml，还是一样无法解决问题。最后，参考了点击打开链接  ，评论中的

所示，终于搞定！

解决Hadoop运行jar包时一直卡在: INFO mapreduce.Job: Running job位置的问题
weixin_44177758的博客
05-06 7978
今天在Hadoop集群运行jar包时一直卡在INFO mapreduce.Job: Running job这个位置，于是上网查了一下，找到了一些解决方案： 下面是卡住都得位置 ： 首先是在我在yarn-site.xml中加 <property> <name>yarn.nodemanager.resource.memory-mb</name> &...
Hive调用MapReduce任务失败
sky_flying1的博客
09-24 5285
1. Hive简介Hive（蜂巢）Apache Hadoop生态圈的构成之一，其主要功能是基于Hadoop提供MapReduce的类SQL查询。Hive的语法规则和Mysql中SQL的语法规则极为相似，有Mysql使用经验的同学都能轻松上手。 Hive的执行分为本地模式和集群模式。本地模式执行简单的Hive SQL，不需要启动MapReduce程序，如-- 本地模式 select col_name
Hadoop运行mapreduce卡住或者一直停在map 0% reduce 0%的解决方案
热门推荐
家里没矿
10-20 2万+
首先先要说明我的系统的一个配置 主机名 主机IPV4地址 配置 hadoop001 192.168.120.100 RecourceManager、NodeManager、DateNode、NameNode hadoop002 192.168.120.101 NodeManager、DateNode、SecondaryNameNode hadoop003 192.168.1...
关于Hadoop下运行MapReduce任务卡在Map0%和Reduce0%的解决办法
_Ashore的博客
05-25 6152
关于Hadoop下运行MapReduce任务卡在Map0%和Reduce0%的解决办法 检查核心配置 mapred-site.xml文件 检查以下配置信息，如果没有该信息，则添加上去 ![在这里插入图片描述](https://img-blog.csdnimg.cn/20200525163616428.png 放个对比图吧（这里的端口不一定要和我的一样，只要不冲突的都可以） 在这里插入代码片 <property> <name>mapreduce.framework.name</
mapreduce 报错
理性的浪漫的程序人生
08-02 877
[root@master hadoop-2.6.5]# hadoop jar /opt/hadoop-mapreduce-examples-2.6.5.jar wordcount /user/root/ds1.txt /user/root/ds_out 17/07/28 02:50:53 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.88.68:8032 17/07/28 02:50:54 INFO input.Fi
执行mapreduce没报错，可是任务运行到running job就卡住在INFO mapreduce.Job: Running job: job_1653745295904_0001
ssn520的博客
05-28 915
执行mapreduce没报错，可是任务运行到runningjob就卡住在INFO mapreduce.Job: Running job: job_1653745295904_0001 使用官方部署的正则进行测试 bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar grep input output 'dfs[a-z.]+' 也是一样，运行到INFO mapreduce.Job: Running job:.
sqoop导hive数据到mysql卡死在INFO mapreduce.Job: map 100% reduce 0%
大黄_sama
02-22 3173
       用sqoop将hive里面的数据导出到mysql里去，一种情况是mapreduce 程序卡住，INFO mapreduce.Job: map 100% reduce 0%       另一种情况是：报这个错误：java.sql.SQLException: Incorrect string value: '\xF0\x9F\x90\xBB' for column       经...
2025-11-19 03:15:50,086 INFO mapreduce.Job: map 0% reduce 0% 2025-11-19 03:16:03,410 INFO mapreduce.Job: map 100% reduce 0%如何终止运行
最新发布
11-20
<name>yarn.app.mapreduce.am.job.timeout <value>600000</value> <!-- 10分钟超时 --> ``` 2. **资源监控脚本**： ```bash #!/bin/bash APP_ID=$(yarn application -list | grep "RUNNING" | awk '{print ...
[root@master exp3]# hadoop jar /export/servers/hadoop-3.4.0/share/hadoop/tools/lib/hadoop-streaming-3.4.0.jar -D mapreduce.job.reduces=1 -D stream.map.output.field.separator=\t -D stream.num.map .output.key.fields=1 -mapper "python /export/data/chap-5/exp3/Mapper.py" -reducer "python /export/data/chap-5/exp3/Reducer.py" -input /exp1/input -output /exp1/output_$(date +%s) -file ./Mapper.py -file ./Reducer.py 2025-05-31 16:24:06,054 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead. packageJobJar: [./Mapper.py, ./Reducer.py, /tmp/hadoop-unjar2796039736802025778/] [] /tmp/streamjob2856802959225773798.jar tmpDir=null 2025-05-31 16:24:06,522 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.24.67:8032 2025-05-31 16:24:06,621 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.24.67:8032 2025-05-31 16:24:06,844 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1748671547381_0012 2025-05-31 16:24:07,087 INFO mapred.FileInputFormat: Total input files to process : 1 2025-05-31 16:24:07,123 INFO mapreduce.JobSubmitter: number of splits:2 2025-05-31 16:24:07,218 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1748671547381_0012 2025-05-31 16:24:07,218 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-05-31 16:24:07,334 INFO conf.Configuration: resource-types.xml not found 2025-05-31 16:24:07,334 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-05-31 16:24:07,379 INFO impl.YarnClientImpl: Submitted application application_1748671547381_0012 2025-05-31 16:24:07,404 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1748671547381_0012/ 2025-05-31 16:24:07,405 INFO mapreduce.Job: Running job: job_1748671547381_0012 2025-05-31 16:24:13,507 INFO mapreduce.Job: Job job_1748671547381_0012 running in uber mode : false 2025-05-31 16:24:13,508 INFO mapreduce.Job: map 0% reduce 0% 2025-05-31 16:24:19,780 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:19,816 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:23,906 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:23,912 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:27,978 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000001_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:29,025 INFO mapreduce.Job: Task Id : attempt_1748671547381_0012_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:326) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:539) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:129) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:466) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) 2025-05-31 16:24:33,127 INFO mapreduce.Job: map 50% reduce 100% 2025-05-31 16:24:34,137 INFO mapreduce.Job: map 100% reduce 100% 2025-05-31 16:24:34,145 INFO mapreduce.Job: Job job_1748671547381_0012 failed with state FAILED due to: Task failed task_1748671547381_0012_m_000001 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-05-31 16:24:34,197 INFO mapreduce.Job: Counters: 14 Job Counters Failed map tasks=7 Killed map tasks=1 Killed reduce tasks=1 Launched map tasks=8 Other local map tasks=6 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=23810 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=23810 Total vcore-milliseconds taken by all map tasks=23810 Total megabyte-milliseconds taken by all map tasks=24381440 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 2025-05-31 16:24:34,197 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!
06-01
我需要回顾之前的回答，用户的问题是关于输出为空，解决方案涉及输入路径验证、权限修复、脚本调试等。现在的新问题涉及子进程失败，可能与脚本错误、环境配置、资源权限等相关。需要整合这些信息，给出分步骤的解决...
解决：Sqoop导入数据到Hive连接hiveserver2卡住
我的博客
07-27 3278
问题 执行sqoop job要将mysql中数据导入hive表，执行到连接hiveserver2时就卡住不动 单独执行连接hiveserver2可以正常连接 Import options: sqoop import --connect jdbc:mysql://xxx:3306/test --username xxxxx --password xxxxxx --table t_user --fields-terminated-by '\001' --hive-import --hive-data
写MapReduce程序时遇到错误Error: INFO - Job job_local1832672383_0001 failed with state FAILED due to: NA
weixin_54527017的博客
05-26 5069
写MapReduce程序时遇到错误Error:INFO - Job job_local1832672383_0001 failed with state FAILED due to: NA 查看发现有错误 第一次遇到这个问题时候，我查了很久没有查出来~ 网上的资料也很少所以便记录一下这次的bug！！其实原理很简单，我自己类型弄错了哭死~~~ 找寻方法如下~~ 需要重新查看自己所输入的类型是否一致！！ 因此将IntWritable 改为LongWritable就可以了 下图为我的Mapp.
Hadoop运行任务时一直卡在: INFO mapreduce.Job: Running job以及Hive执行也卡住的情况
zhangxu1001的博客
09-02 2185
在机器的配置没有修改的情况下，出现这样的问题，后来经过多方排查，将每台机器上配置的fs的默认临时文件目录全部删除之后，再重启yarn得以解决，使用yarn node -list -all可以查看到此时的节点的状态就不是unhealthy了，变成了running。 ...
MapReduce执行任务时卡在mapreduce.Job: Running Job这一步该如何解决
sinat_33769106的博客
07-03 1万+
问题描述：在centos7系统搭建的伪分布式Hadoop基础上，用Sqoop把HDFS上的数据导出到Mysql的时候，发现任务执行时卡死在如下图所示的这一步：卡了接近十分钟意识到可能永远也不会结束了....赶紧ctrl+c结束进程，找找原因。首先搜了下资料，发现大多数教程都说是内存不足引起的，可在运行这个命令之前只运行过一次wordcount例程，按道理不可能占用过多内存。排除内存问题。打开had...
Hive 任务卡在 map = 0%, reduce = 0%
Alex
06-21 4054
Hive 卡在map = 0%, reduce = 0%阶段 解决：增加map个数，设置mapreduce.input.fileinputformat.split.maxsize 小于系统默认值，需要综合考虑调用更多map时候的消耗 定位时发现任务卡在map = 0%, reduce = 0%阶段，而且发现map分配数量很少，所以猜测分配map较慢，增加map数量可解决问题。 ...
job全异常 mapreduce_Sqoop导入数据异常处理
weixin_39945679的博客
02-01 298
1.错误提示(没有json.jar包)19/01/30 11:59:48 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import19/01/30 11:59:48 INFO mapreduce.ImportJobBase: Beginning import of t3Exception in thread "mai...
hadoop相关问题
weixin_41347226的博客
06-05 904
一、mapreduce作业oom 1、任务还没启动,直接oom报错 AM日志： 21/05/10 15:15:13 INFO mapreduce.Job: Task Id : attempt_1617064346277_101596_m_000000_1, Status : FAILEDError: Java heap space21/05/10 15:15:16 INFO mapreduce.Job: Task Id : attempt_1617064346277_101596_m_000000_2, S
hadoop job java_java编译通过，但是在hadoop上运行卡在jobrunning，然后报错timeout，是为什么呢？...
weixin_32620249的博客
02-23 803
hadoop jar load.jar caivbslist /user/ip/public/cbill_ticket_cdma_voice/201608/20160805 /user/ip/public/cbill_ticket_ivpn/201608/20160805 /user/ffcs_sales/private/output/load_interface/2016080...
Hadoop运行job程序报错 exitCode=255
u011360625的博客
02-22 1万+
Hadoop版本：Hadoop-2.6.5 运行环境：三台虚拟主机，主机名分别是 duanyd、hadoop_01、hadoop_02 刚学hadoop，照着老师的示例编了个统计词数的mapredurce程序，传到duanyd虚拟主机运行后报如下错误： 17/02/22 06:11:13 INFO mapreduce.Job: Job job_1487677042735_0006 faile
Hadoop写压测试出现Job failed!
ggssc的博客
02-07 832
hadoop写压测试，集群性能不够会如何
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值