报错:
running beyondphysical memory limits.Current usage: 1.3 GB of 1.7 GBphysical memory used; 7.4 GB of 13.2 TB virtual memory used. Killing container。
解决方案
# 为MapReduce增大堆内存
set mpareduce.map.java.opts=-Xmx3072 -XX:++UseConcMarkSweepGC;
set mapreduce.map.memory.mb=4096;
set mapreduce.reduce.java.opts=-Xmx3072m -XX:+UseConcMarkSweepGC;
set mapreduce.reduce.memory.mb=4096;
如果设置完之后仍然报错,可能是map切片过大或者reduce数量不够,增加以下设置
set mapred.max.split.size=128000000;
set mapred.reduce.tasks=60;