nutch 0.9二次开发--内存溢出(转)

最新推荐文章于 2025-08-17 21:38:20 发布

转载最新推荐文章于 2025-08-17 21:38:20 发布 · 84 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/Myhsg/archive/2010/03/05/1678839.html

文章标签：

#java #大数据

本文介绍了解决Nutch在抓取网页过程中遇到的内存溢出问题。通过调整Hadoop配置文件中的Java堆内存大小或者增大Java虚拟机的堆内存，可以有效避免该问题的发生。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

原文地址： http://blog.chinaunix.net/u2/73798/showart_1731875.html

在用nutch抓取网页的时候，设置了10层，运行5个多小时之后，系统提示内存溢出异常:

java.lang.OutOfMemoryError: Java heap space
fetcher caught:java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
fetcher caught:java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:470)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)

问题分析:

解决办法1：

Add the following to hadoop-site.xml. This sets the Java heap size for the
spawned child process. You can set it to whatever you want. I believe the
default size is 200 MB which is way too small.

<property>
<name>mapred.child.java.opts</name>
<value>-Xmx512m</value>
</property>

解决办法2：

increase the heapsize available to your java VM. Do this using the Run->"Open Run Dialog" menu, clicking the arguments tab and setting the vm arguments to include -Xmx1024m (or something larger, if need be). Obviously, it doesn't really make sense to run jobs over huge datasets on your puny little laptop. You should debug and develop only on a small portion of the actual data.

相关资料:

Re: java.lang.OutOfMemoryError: Java heap space

>
>>
>> On Thu, Sep 18, 2008 at 4:19 PM, Edward Quick < edwardquick@...> wrote:
>> >
>> > Hi,
>> >
>> > I'm getting java.lang.OutOfMemoryError: Java heap space errors when running nutch in a hadoop cluster.
>> > I have doubled the heap by setting export HADOOP_HEAPSIZE=2048 in hadoop-env.sh but this doesn't seem to make a difference.
>> >
>> > I'm need to hadoop so appreciate any help.
>> >
>>
>> Are you parsing during fetching? If so try disabling that and run
>> parsing as a separate job. At least, you
>> won't lose the results of fetching :)
>
>
> The threads in nutch-site.xml were set too high (at 50) so I put those down to 10 and it seems ok now.
>

转载于:https://www.cnblogs.com/Myhsg/archive/2010/03/05/1678839.html