lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "McKinley, James T" <james.mckin...@cengage.com>
Subject RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)
Date Fri, 06 Feb 2015 17:10:31 GMT
Just to be clear in case there was any confusion about my previous message regarding G1GC,
we do not use Solr, my team works on a proprietary Lucene-based search engine.  Consequently,
I can't really give any advice regarding Solr with G1GC, but for our uses (so far anyway),
G1GC seems to work well with Lucene.

Jim
________________________________________
From: Piotr Idzikowski [piotridzikowski@gmail.com]
Sent: Friday, February 06, 2015 5:35 AM
To: java-user@lucene.apache.org
Subject: Re: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)

Hello.
A little bit delayed question. But recently I have found this articles:
https://wiki.apache.org/solr/SolrPerformanceProblems
https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

Especially this part from first url:
*Using the ConcurrentMarkSweep (CMS) collector with tuning parameters is a
very good option for for Solr, but with the latest Java 7 releases (7u72 at
the time of this writing), G1 is looking like a better option, if the
-XX:+ParallelRefProcEnabled option is used.*

How does it play with *"Do not, under any circumstances, run Lucene with
the G1 garbage collector."*
from https://wiki.apache.org/lucene-java/JavaBugs?

Regards
Piotr

On Tue, Jan 27, 2015 at 9:55 PM, McKinley, James T <
james.mckinley@cengage.com> wrote:

> Hi Uwe,
>
> OK, thanks for the info.  We'll see if we can download the Lucene test
> suite and check it out.
>
> FWIW, we use G1GC in our production runtime (~70 12-16 core Cisco UCS and
> HP Gen7/Gen8 nodes with 20+ GB heaps using Java 7 and Lucene 4.8.1 with
> pairs of 30 index partitions with 15M-23M docs each) and have not
> experienced any VM crashes (well, maybe a couple, but not directly
> traceable to G1 to my knowledge).  We have found some undocumented pauses
> in G1 due to very large object arrays and filed a bug report which was
> confirmed and also affects CMS (we worked around this in our code using
> memory mapping of some files whose contents we previously held all in
> RAM).  I think the only index corruption we've ever seen was in our index
> creation workflow (~30 HP Gen7 nodes with 27GB heaps) but this was using
> Parallel GC since it is a batch system, so that corruption (which we've not
> seen recently and never found a cause for) was definitely not due to G1GC.
>
> G1GC has bugs as does CMS but we've found it to work pretty well so far in
> our runtime system.  Of course YMMV, thanks again for the info.
>
> Jim
> ________________________________________
> From: Uwe Schindler [uwe@thetaphi.de]
> Sent: Tuesday, January 27, 2015 3:02 PM
> To: java-user@lucene.apache.org
> Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)
>
> Hi.,
>
> About G1GC. We consistently see problems when running the Lucene Testsuite
> with G1GC enabled. The people from Elasticsearch concluded:
>
> "There is a newer GC called the Garbage First GC (G1GC). This newer GC is
> designed to minimize pausing even more than CMS, and operate on large
> heaps. It works by dividing the heap into regions and predicting which
> regions contain the most reclaimable space. By collecting those regions
> first (garbage first), it can minimize pauses and operate on very large
> heaps.
>
> Sounds great! Unfortunately, G1GC is still new, and fresh bugs are found
> routinely. These bugs are usually of the segfault variety, and will cause
> hard crashes. The Lucene test suite is brutal on GC algorithms, and it
> seems that G1GC hasn’t had the kinks worked out yet.
>
> We would like to recommend G1GC someday, but for now, it is simply not
> stable enough to meet the demands of Elasticsearch and Lucene."
> (
> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_don_8217_t_touch_these_settings.html
> )
>
> In fact, the problems with G1GC can sometimes lead to index corruption,
> and are hard to reproduce. So better don't use...
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: McKinley, James T [mailto:james.mckinley@cengage.com]
> > Sent: Tuesday, January 27, 2015 8:58 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)
> >
> > Why do you say not to use G1GC?  We are using Java 7 & G1GC with Lucene
> > 4.8.1 in production.  Thanks.
> >
> > Jim
> > ________________________________________
> > From: Uwe Schindler [uwe@thetaphi.de]
> > Sent: Tuesday, January 27, 2015 2:49 PM
> > To: java-user@lucene.apache.org; 'kiwi clive'
> > Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)
> >
> > Java 8 update 20 or later is also fine. At current time, always use
> latest update
> > release and you are be fine with Java 7 and Java 8. Don't use older
> releases
> > and don't use G1 Garbage Collector.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: kiwi clive [mailto:kiwi_clive@yahoo.com.INVALID]
> > > Sent: Tuesday, January 27, 2015 8:03 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8)
> > >
> > > Hi Hoss,
> > > Many thanks for the information. This looks very encouraging as the
> > > Java7 bug I remember  was fixed and as far as I know, we should not be
> > > affected by the others.
> > > I'll put a few tests together and put my toe in the water :-) Clive
> > >
> > >       From: Chris Hostetter <hossman_lucene@fucit.org>
> > >  To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>; kiwi
> > > clive <kiwi_clive@yahoo.com>
> > >  Sent: Tuesday, January 27, 2015 4:01 PM
> > >  Subject: Re: Lucene Version Upgrade (3->4) and Java JVM
> > > Versions(6->8)
> > >
> > >
> > >
> > >
> > > : I seem to remember reading that certain versions of lucene were
> > > : incompatible with some java versions although I cannot find anything
> > > to
> > > : verify this. As we have tens of thousands of large indexes,
> > > backwards
> > > : compatibility without the need to reindex on an upgrade is of prime
> > > : importance to us.
> > >
> > > All known JVM bugs affecting Lucene are listed here...
> > >
> > > https://wiki.apache.org/lucene-java/JavaBugs
> > >
> > >
> > > -Hoss
> > > http://www.lucidworks.com/
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message