lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jena <sthita2...@gmail.com>
Subject Re: Urgent help on solr optimisation issue !!
Date Fri, 07 Jun 2019 15:21:31 GMT
Thanks @Michael Joyner,  how did you decide initialmax segment to 256 ? Or it
is some random number i can use for my case ? Can you guuide me how to
decide the initial & final max segments ?

 
Michael Joyner wrote
> That is the way we do it here - also helps a lot with not needing x2 or 
> x3 disk space to handle the merge:
> 
> public void solrOptimize() {
>          int initialMaxSegments = 256;
>          int finalMaxSegments = 4;
>          if (isShowSegmentCounter()) {
>              log.info("Optimizing ...");
>          }
>          try (SolrClient solrServerInstance = getSolrClientInstance()) {
>              for (int segments = initialMaxSegments; segments >= 
> finalMaxSegments; segments--) {
>                  if (isShowSegmentCounter()) {
>                      System.out.println("Optimizing to a max of " + 
> segments + " segments.");
>                  }
>                  try {
>                      solrServerInstance.optimize(true, true, segments);
>                  } catch (RemoteSolrException | SolrServerException | 
> IOException e) {
>                      log.severe(e.getMessage());
>                  }
>              }
>          } catch (IOException e) {
>              throw new RuntimeException(e);
>          }
>      }
> 
> On 6/7/19 4:56 AM, Nicolas Franck wrote:
>> In that case, hard optimisation like that is out the question.
>> Resort to automatic merge policies, specifying a maximum
>> amount of segments. Solr is created with multiple segments
>> in mind. Hard optimisation seems like not worth the problem.
>>
>> The problem is this: the less segments you specify during
>> during an optimisation, the longer it will take, because it has to read
>> all of these segments to be merged, and redo the sorting. And a cluster
>> has a lot of housekeeping on top of it.
>>
>> If you really want to issue a optimisation, then you can
>> also do it in steps (max segments parameter)
>>
>> 10 -> 9 -> 8 -> 7 .. -> 1
>>
>> that way less segments need to be merged in one go.
>>
>> testing your index will show you what a good maximum
>> amount of segments is for your index.
>>
>>> On 7 Jun 2019, at 07:27, jena &lt;

> sthita2010@

> &gt; wrote:
>>>
>>> Hello guys,
>>>
>>> We have 4 solr(version 4.4) instance on production environment, which
>>> are
>>> linked/associated with zookeeper for replication. We do heavy deleted &
>>> add
>>> operations. We have around 26million records and the index size is
>>> around
>>> 70GB. We serve 100k+ requests per day.
>>>
>>>
>>> Because of heavy indexing & deletion, we optimise solr instance
>>> everyday,
>>> because of that our solr cloud getting unstable , every solr instance go
>>> on
>>> recovery mode & our search is getting affected & very slow because of
>>> that.
>>> Optimisation takes around 1hr 30minutes.
>>> We are not able fix this issue, please help.
>>>
>>> Thanks & Regards
>>>
>>>
>>>
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Mime
View raw message