lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Improvement performance of my indexing with Lucene
Date Wed, 09 Sep 2015 17:00:02 GMT
> Great! I will upgrade Lucene then.

Good start.

> I'm not using database.

Fine, but you must be getting your data from somewhere.  Maybe that is
blazingly fast, maybe it isn't.

> Are there some java samples code ?
>
> Samples with:
>
> 1. indexing documents in batches.

I think this means call IndexWriter.commit() every some-large-number
of docs rather than some-small-number.

> 2. Multi-threaded indexing

I don't have examples, but pseudocode would look something like

 IndexWriter iw = whatever
 Thread t1 = whatever(iw, data-source-1)
 Thread t2 = whatever(iw, data-source-2)
 ...
 t1.start()
 t2.start()
 ...
 wait ...
 iw.close()


--
Ian.


> On Wed, Sep 9, 2015 at 11:23 AM, Ian Lea <ian.lea@gmail.com> wrote:
>
>> The link that I sent,
>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed is for Lucene,
>> not Solr.  The second item on the list is to make sure you are using
>> the latest version of lucene so that would be a good starting point.
>>
>>
>> --
>> Ian.
>>
>>
>> On Wed, Sep 9, 2015 at 3:10 PM, Humberto Rocha <humrocha@gmail.com> wrote:
>> > Thanks a lot !
>> >
>> > But do you know some links that helps implement these optimization
>> options
>> > without the Solr (using only lucene) ?
>> >
>> > I am using lucene 4.9.
>> >
>> > More thanks.
>> >
>> > Humberto
>> >
>> >
>> > On Wed, Sep 9, 2015 at 5:23 AM, Ian Lea <ian.lea@gmail.com> wrote:
>> >
>> >> See also http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>> >>
>> >> Also double check that it's Lucene that you should be concentrating
>> >> on.  In my experience it's often the reading of the data from a
>> >> database, if that's what you are doing, that is the bottleneck.
>> >>
>> >>
>> >> --
>> >> Ian.
>> >>
>> >>
>> >> On Wed, Sep 9, 2015 at 6:07 AM, Modassar Ather <modather1981@gmail.com>
>> >> wrote:
>> >> > There are few things you can try to improve indexing performance.
>> >> >
>> >> > 1. Try indexing documents in batches.
>> >> > 2. You can try multi-threaded indexing. What I mean to say is feed
the
>> >> data
>> >> > using multiple threads to the indexer.
>> >> > 3. Analysis of memory utilization and GC tuning.
>> >> >
>> >> > Following are few links which has few details on Solr indexing
>> >> performance.
>> >> > http://wiki.apache.org/solr/SolrPerformanceFactors
>> >> >
>> >>
>> https://lucidworks.com/blog/indexing-performance-solr-5-2-now-twice-fast/
>> >> >
>> >> > Regards,
>> >> > Modassar
>> >> >
>> >> > On Wed, Sep 9, 2015 at 7:29 AM, Humberto Rocha <humrocha@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I need to improve the performance of my indexing with Lucene .
>> >> >>
>> >> >> Is there any material (eg, article, book , tutorial ) that can
be
>> used
>> >> for
>> >> >> this?
>> >> >>
>> >> >> Could anyone help me please ?
>> >> >>
>> >> >> Thanks a lot!
>> >> >>
>> >> >> --
>> >> >> Humberto
>> >> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > Humberto Rocha
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Humberto Rocha

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message