hadoop 解决内存溢出问题 2.1 GB of 2.1 GB virtual memory used. Killing container

hadoop集群启动成功之后,牛刀小试一下自带的jar包里的程序,计算一下pi值:

报如下错误:2.1 GB of 2.1 GB virtual memory used. Killing container

1.在etc/hadoop/yarn-site.xml文件中,修改检查虚拟内存的属性为false,如下:

 

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

运行一下还是报同样的错误

再次添加如下内容:

2.修改mapred-site.xml 配置文件,加入如下内容:

<property>
<name>mapreduce.cluster.map.memory.mb</name>
<value>-1</value>
</property>
<property>
<name>mapreduce.cluster.reduce.memory.mb</name>
<value>-1</value>
</property>
 

 

重新启动集群

成功!!计算出pi值

原文参考:https://blog.youkuaiyun.com/ma0903/article/details/48289731

https://blog.youkuaiyun.com/dd390738878/article/details/9785679

 

 

486_0003 running in uber mode : false 2025-06-11 18:48:39,482 INFO mapreduce.Job: map 0% reduce 0% 2025-06-11 18:48:45,635 INFO mapreduce.Job: Task Id : attempt_1744881937486_0003_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2025-06-11 18:48:46,677 INFO mapreduce.Job: map 50% reduce 0% 2025-06-11 18:48:50,727 INFO mapreduce.Job: Task Id : attempt_1744881937486_0003_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2025-06-11 18:48:52,758 INFO mapreduce.Job: Task Id : attempt_1744881937486_0003_r_000000_0, Status : FAILED [2025-06-11 18:48:51.419]Container [pid=71601,containerID=container_1744881937486_0003_01_000005] is running 537561600B beyond the &#39;VIRTUAL&#39; memory limit. Current usage: 166.6 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1744881937486_0003_01_000005 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 71601 71599 71601 71601 (bash) 0 1 116002816 303 /bin/bash -c /usr/local/jdk1.8.0_391/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1744881937486_0003/container_1744881937486_0003_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/software/hadoop-3.1.3/logs/userlogs/application_1744881937486_0003/container_1744881937486_0003_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.30.110 37705 attempt_1744881937486_0003_r_000000_0 5 1>/root/software/hadoop-3.1.3/logs/userlogs/application_1744881937486_0003/container_1744881937486_0003_01_000005/stdout 2>/root/software/hadoop-3.1.3/logs/userlogs/application_1744881937486_0003/container_1744881937486_0003_01_000005/stderr |- 71616 71601 71601 71601 (java) 453 87 2676416512 42355 /usr/local/jdk1.8.0_391/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1744881937486_0003/container_1744881937486_0003_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/software/hadoop-3.1.3/logs/userlogs/application_1744881937486_0003/container_1744881937486_0003_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.30.110 37705 attempt_1744881937486_0003_r_000000_0 5 [2025-06-11 18:48:51.462]Container killed on request. Exit code is 143 [2025-06-11 18:48:51.478]Container exited with a non-zero exit code 143. 2025-06-11 18:48:54,802 INFO mapreduce.Job: Task Id : attempt_1744881937486_0003_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2025-06-11 18:49:00,976 INFO mapreduce.Job: map 100% reduce 100% 2025-06-11 18:49:01,992 INFO mapreduce.Job: Job job_1744881937486_0003 failed with state FAILED due to: Task failed task_1744881937486_0003_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-06-11 18:49:02,105 INFO mapreduce.Job: Counters: 42 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=352125 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1771275 HDFS: Number of bytes written=0 HDFS: Number of read operations=3 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Failed map tasks=4 Failed reduce tasks=1 Killed reduce tasks=1 Launched map tasks=5 Launched reduce tasks=2 Other local map tasks=3 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=18987 Total time spent by all reduces in occupied slots (ms)=8718 Total time spent by all map tasks (ms)=18987 Total time spent by all reduce tasks (ms)=8718 Total vcore-milliseconds taken by all map tasks=18987 Total vcore-milliseconds taken by all reduce tasks=8718 Total megabyte-milliseconds taken by all map tasks=19442688 Total megabyte-milliseconds taken by all reduce tasks=8927232 Map-Reduce Framework Map input records=10009 Map output records=10009 Map output bytes=110099 Map output materialized bytes=130123 Input split bytes=92 Combine input records=0 Spilled Records=10009 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=121 CPU time spent (ms)=1250 Physical memory (bytes) snapshot=268398592 Virtual memory (bytes) snapshot=2787106816 Total committed heap usage (bytes)=249036800 Peak Map Physical memory (bytes)=268398592 Peak Map Virtual memory (bytes)=2787106816 File Input Format Counters Bytes Read=1771183 2025-06-11 18:49:02,105 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed! [root@master ~]#
最新发布
06-12
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值