lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Several builds hanging pecause of permgen
Date Tue, 09 Jul 2013 07:32:23 GMT
I inspected the heap dumps already, 2 things are interesting:

- The heap dump file is quite small (only approx. 50 MB), so the permgen issues start when
not even -Xmx is used completely.
- There are lots of (250) reflection class loaders, but no leaks. All classloaders exist one
time, just the reflection-based sun.relect.DelegatingClassLoader with the stubs for speeding
up reflection shows up, but only with one class/loader. Which is perfectly fine.
- ~2500 classes loaded, which is not much.

How do I get the number of interned Strings from jvisualvm?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On Behalf
> Of Dawid Weiss
> Sent: Tuesday, July 09, 2013 9:13 AM
> To: dev@lucene.apache.org
> Subject: Re: Several builds hanging pecause of permgen
> 
> What's the core of those permgen leaks, guys? Is it the number of classes
> (via different classloaders)? Number of interned strings?
> Maybe something is going on that we should reflect on before we raise the
> default permgen limit? Our typical permgen headaches were due to
> repeated classloader leaks -- is Hadoop using something like that?
> 
> Dawid
> 
> On Tue, Jul 9, 2013 at 8:25 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
> > Hi Mark,
> >
> > The problem with raising permgen is:
> > - It's Hotspot specific only, so does not work with other JVMs
> > - Its no longer available in Java 8
> >
> > I would really prefer to maybe tune the tests and maybe not create so
> many nodes in the cloud tests. It looks like the bug happens more often with
> higher test multiplier (-Dtests.multiplier=3), so maybe we can really tune
> that.
> > If we want to raise permgen, we have to do it in a similar way like we
> > do enable the heap dumps - with lots of <condition/> tasks in ANT...
> > :(
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Mark Miller [mailto:markrmiller@gmail.com]
> >> Sent: Tuesday, July 09, 2013 5:20 AM
> >> To: dev@lucene.apache.org
> >> Subject: Re: Several builds hanging pecause of permgen
> >>
> >> Looks like we currently don't set the max perm gen for tests, so you
> >> get the default - I think we want to change that regardless - we
> >> don't want it to vary IMO - it should work like Xmx.
> >>
> >> I think we should just set it to 128 mb, and these tests should have
> >> plenty of room to run.
> >>
> >> - Mark
> >>
> >> On Jul 8, 2013, at 11:06 PM, Mark Miller <markrmiller@gmail.com> wrote:
> >>
> >> >
> >> > On Jul 8, 2013, at 2:26 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> >> >
> >> >> Next one:
> >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6395/console
> >> >>
> >> >> I will vote all releases of 4.4 with -1 until this is fixed!
> >> >
> >> > You can't veto a release, so it's kind of a hollow threat ;)
> >> >
> >> >> It hangs on my local computer, too! Tests pass only ½ of the time,
> >> >> the
> >> remaining time it hangs with permgen errors.
> >> >
> >> > Depending on the host OS and java version, I have had to raise the
> >> > max
> >> perm gem a bit higher for heavy SolrCloud tests that also start up
> >> hdfs. I think I remember raising it from 96 MB to 128 MB. It's simply
> >> a test resource issue - those tests are very heavy, which is why most
> >> are set to run nightly - I've seen the code run heavily without any
> >> perm gen issues though - and on some of my machines, I don't have to
> raise the perm gen at all.
> >> >
> >> > - Mark
> >> >
> >> >>
> >> >> Uwe
> >> >>
> >> >> -----
> >> >> Uwe Schindler
> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> >> eMail: uwe@thetaphi.de
> >> >>
> >> >> From: dawid.weiss@gmail.com [mailto:dawid.weiss@gmail.com] On
> >> Behalf
> >> >> Of Dawid Weiss
> >> >> Sent: Monday, July 08, 2013 2:16 PM
> >> >> To: dev@lucene.apache.org
> >> >> Subject: Re: Several builds hanging pecause of permgen
> >> >>
> >> >>
> >> >> Not much I can do from my side about permgen errors. There is
> >> >> really no
> >> way to deal with these from within Java (the same process) -- you
> >> cannot effectively handle anything because your own classes may not
> load at all.
> >> >>
> >> >> Dawid
> >> >>
> >> >> On Mon, Jul 8, 2013 at 1:35 PM, Uwe Schindler <uwe@thetaphi.de>
> >> wrote:
> >> >> Another one, this time on OSX:
> >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/617/
> >> >>
> >> >> -----
> >> >> Uwe Schindler
> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> >> eMail: uwe@thetaphi.de
> >> >>
> >> >> From: Robert Muir [mailto:rcmuir@gmail.com]
> >> >> Sent: Sunday, July 07, 2013 11:15 PM
> >> >> To: dev@lucene.apache.org
> >> >> Subject: Re: Several builds hanging pecause of permgen
> >> >>
> >> >> When there were leaks from static classes, we added a checker to
> >> LuceneTestCase that looks for RAM > N and fails with debugging
> information.
> >> >>
> >> >> I wonder if some similar check is possible for this case (to make
> >> >> it easier
> >> than going thru heapdumps, and to find issues before crash-time)...
> >> >>
> >> >> On Sun, Jul 7, 2013 at 4:10 PM, Uwe Schindler <uwe@thetaphi.de>
> wrote:
> >> >> Another one:
> >> >> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6385/testRepo
> >> >> rt/
> >> >>
> >>
> junit/junit.framework/TestSuite/org_apache_solr_request_SimpleFacetsT
> >> >> est/
> >> >>
> >> >> Had to be killed with kill -9
> >> >>
> >> >> -----
> >> >> Uwe Schindler
> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> >> eMail: uwe@thetaphi.de
> >> >>
> >> >>
> >> >>> -----Original Message-----
> >> >>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> >> >>> Sent: Saturday, July 06, 2013 10:16 PM
> >> >>> To: dev@lucene.apache.org
> >> >>> Subject: RE: Several builds hanging pecause of permgen
> >> >>>
> >> >>> Another one:
> >> >>> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6375/console
> >> >>>
> >> >>> I was only able to kill the JVM with kill -9 I am sure, it's horrible
> >> slowdoop!
> >> >>>
> >> >>> -----
> >> >>> Uwe Schindler
> >> >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> >>> eMail: uwe@thetaphi.de
> >> >>>
> >> >>>
> >> >>>> -----Original Message-----
> >> >>>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> >> >>>> Sent: Friday, July 05, 2013 3:59 PM
> >> >>>> To: dev@lucene.apache.org
> >> >>>> Subject: Several builds hanging pecause of permgen
> >> >>>>
> >> >>>> Several Jenkins builds now hang because of permgen. The runner
> JVM
> >> >>>> is dead (can only be killed by -9), last example:
> >> >>>>
> >> >>>> http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6360/console
> >> >>>>
> >> >>>> -----
> >> >>>> Uwe Schindler
> >> >>>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >> >>>> eMail: uwe@thetaphi.de
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> -------------------------------------------------------------------
> >> >>>> -- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For
> >> >>>> additional commands, e-mail: dev-help@lucene.apache.org
> >> >>>
> >> >>>
> >> >>> --------------------------------------------------------------------
> >> >>> - To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> >> >>> additional commands, e-mail: dev-help@lucene.apache.org
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> >> >> additional commands, e-mail: dev-help@lucene.apache.org
> >> >>
> >> >>
> >> >
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> additional
> >> commands, e-mail: dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message