运行mahout 将学习集的序列化转化为向量报错Error: Java heap space(mapreduce运行内存调优)

最新推荐文章于 2024-09-16 09:00:00 发布

原创最新推荐文章于 2024-09-16 09:00:00 发布 · 1.1k 阅读

0 ·

CC 4.0 BY-SA版权

hadoop 同时被 2 个专栏收录

16 篇文章

订阅专栏

mahout

4 篇文章

订阅专栏

mahout seq2sparse -i ./email-seq -o ./email-vectors  -lnorm -nv  -wt tfidf


15/10/28 09:36:58 INFO FileInputFormat: Total input paths to process : 1
15/10/28 09:36:58 INFO JobSubmitter: number of splits:1
15/10/28 09:36:58 INFO JobSubmitter: Submitting tokens for job: job_1445824701143_0007
15/10/28 09:36:59 INFO YarnClientImpl: Submitted application application_1445824701143_0007
15/10/28 09:36:59 INFO Job: The url to track the job: http://hcg1:8088/proxy/application_1445824701143_0007/
15/10/28 09:36:59 INFO Job: Running job: job_1445824701143_0007
15/10/28 09:37:10 INFO Job: Job job_1445824701143_0007 running in uber mode : false
15/10/28 09:37:10 INFO Job:  map 0% reduce 0%
15/10/28 09:37:38 INFO Job:  map 100% reduce 0%
15/10/28 09:37:58 INFO Job:  map 100% reduce 100%
15/10/28 09:38:21 INFO Job: Task Id : attempt_1445824701143_0007_r_000000_0, Status : FAILED
Error: Java heap space
15/10/28 09:38:22 INFO Job:  map 100% reduce 0%
15/10/28 09:38:41 INFO Job:  map 100% reduce 100%
15/10/28 09:39:03 INFO Job: Task Id : attempt_1445824701143_0007_r_000000_1, Status : FAILED
Error: Java heap space
15/10/28 09:39:04 INFO Job:  map 100% reduce 0%

解决方法：调大的MapReduce中的参数mapreduce.map.java.opts

<属性>
    <名> mapreduce.map.java.opts </姓名>
    <值> -Xmx1024m </值>
  </属性>

继续报错Error: GC overhead limit exceeded

15/10/28 09:22:36 INFO FileInputFormat: Total input paths to process : 1
15/10/28 09:22:36 INFO JobSubmitter: number of splits:1
15/10/28 09:22:36 INFO JobSubmitter: Submitting tokens for job: job_1445824701143_0004
15/10/28 09:22:37 INFO YarnClientImpl: Submitted application application_1445824701143_0004
15/10/28 09:22:37 INFO Job: The url to track the job: http://hcg1:8088/proxy/application_1445824701143_0004/
15/10/28 09:22:37 INFO Job: Running job: job_1445824701143_0004
15/10/28 09:23:08 INFO Job: Job job_1445824701143_0004 running in uber mode : false
15/10/28 09:23:08 INFO Job:  map 0% reduce 0%
15/10/28 09:23:26 INFO Job:  map 100% reduce 0%
15/10/28 09:23:53 INFO Job: Task Id : attempt_1445824701143_0004_r_000000_0, Status : FAILED
Error: GC overhead limit exceeded
15/10/28 09:24:22 INFO Job:  map 100% reduce 100%
15/10/28 09:24:22 INFO Job: Task Id : attempt_1445824701143_0004_r_000000_1, Status : FAILED
Error: GC overhead limit exceeded
15/10/28 09:24:23 INFO Job:  map 100% reduce 0%
15/10/28 09:24:50 INFO Job:  map 100% reduce 100%
15/10/28 09:24:50 INFO Job: Task Id : attempt_1445824701143_0004_r_000000_2, Status : FAILED
Error: GC overhead limit exceeded
15/10/28 09:24:51 INFO Job:  map 100% reduce 0%

经过多次试验，我的数据量需要的配置<属性>
    <名> mapreduce.map.java.opts </姓名>
    <值> -Xmx3000m </值>
  </属性>
<属性>
    <名> mapreduce.reduce.java.opts </姓名>
    <值> -Xmx3000m </值>
  </属性>
将hadoop-env.sh的hadoope HADOOP_HEAPSIZE调大

参考：http://blogs.msdn.com/b/shanyu/archive/2014/07/31/hadoop-yarn-memory-settings-in-hdinsigh.aspx

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html