nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: [jira] [Commented] (NUTCH-356) Plugin repository cache can lead to memory leak
Date Fri, 24 Jan 2014 15:27:48 GMT
Hahahahahahahaha

-----Original message-----
From: Tejas Patil<tejas.patil.cs@gmail.com>
Sent: Friday 24th January 2014 16:20
To: dev@nutch.apache.org
Subject: Fwd: [jira] [Commented] (NUTCH-356) Plugin repository cache can lead to memory leak

Hi Lewis,

I wont be surprised if any user out there gets a cake for this jira ;) .. just like someone
did over a MySQL bug (http://www.youtube.com/watch?v=oAiVsbXVP6k <http://www.youtube.com/watch?v=oAiVsbXVP6k>)

Cheers !!

---------- Forwarded message ----------
From: Lewis John McGibbney (JIRA) <jira@apache.org <mailto:jira@apache.org>>

Date: Fri, Jan 24, 2014 at 7:01 PM
Subject: [jira] [Commented] (NUTCH-356) Plugin repository cache can lead to memory leak
To: tejasp@apache.org <mailto:tejasp@apache.org>

    [ https://issues.apache.org/jira/browse/NUTCH-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880956#comment-13880956
<https://issues.apache.org/jira/browse/NUTCH-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880956#comment-13880956>
]

Lewis John McGibbney commented on NUTCH-356:

--------------------------------------------

Indeed Markus. This one is blast from the past for sure. I dont know if I was born when Nutch
0.8 was trunk ;)

> Plugin repository cache can lead to memory leak

> -----------------------------------------------

>

>                 Key: NUTCH-356

>                 URL: https://issues.apache.org/jira/browse/NUTCH-356 <https://issues.apache.org/jira/browse/NUTCH-356>

>             Project: Nutch

>          Issue Type: Bug

>    Affects Versions: 0.8

>            Reporter: Enrico Triolo

>            Assignee: Markus Jelsma

>             Fix For: 2.3, 1.8

>

>         Attachments: ASF.LICENSE.NOT.GRANTED--NutchTest.java, ASF.LICENSE.NOT.GRANTED--patch.txt,
NUTCH-356-trunk.patch, cache_classes.patch

>

>

> While I was trying to solve a problem I reported a while ago (see Nutch-314), I found
out that actually the problem was related to the plugin cache used in class PluginRepository.java.

> As  I said in Nutch-314, I think I somehow force the way nutch is meant to work, since
I need to frequently submit new urls and append their contents to the index; I dont (and I
cant) have an urls.txt file with all urls Im going to fetch, but I recreate it each time a
new url is submitted.

> Thus,  I think in the majority of times you wont have problems using nutch as-is, since
the problem I found occours only if nutch is used in a way similar to the one I use.

> To simplify your test Im attaching a class that performs something similar to what I
need. It fetches and index some sample urls; to avoid webmasters complaints I left the sample
urls list empty, so you should modify the source code and add some urls.

> Then you only have to run it and watch your memory consumption with top. In my experience
I get an OutOfMemoryException after a couple of minutes, but it clearly depends on your heap
settings and on the plugins you are using (Im using protocol-file|protocol-http|parse-(rss|html|msword|pdf|text)|language-identifier|index-(basic|more)|query-(basic|more|site|url)|urlfilter-regex|summary-basic|scoring-opic).

> The problem is bound to the PluginRepository singleton instance, since it never get released.
It seems that some class maintains a reference to it and this class is never released since
it is cached somewhere in the configuration.

> So I modified the PluginRepositorys get method so that it never uses the cache and always
returns a new instance (you can find the patch in attachment). This way the memory consumption
is always stable and I get no OOM anymore.

> Clearly this is not the solution, since I guess there are many performance issues involved,
but for the moment it works.

--

This message was sent by Atlassian JIRA

(v6.1.5#6160)



Mime
View raw message