lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rishi Easwaran <rishi.easwa...@aol.com>
Subject Re: Solr Cloud reclaiming disk space from deleted documents
Date Mon, 04 May 2015 10:55:21 GMT
Sadly with the size of our complex, spiting and adding more HW is not a viable long term solution.

 I guess the options we have are to run optimize regularly and/or become aggressive in our
merges proactively even before solr cloud gets into this situation.
 
 Thanks,
 Rishi.
 

 

 

-----Original Message-----
From: Gili Nachum <gilinachum@gmail.com>
To: solr-user <solr-user@lucene.apache.org>
Sent: Mon, Apr 27, 2015 4:18 pm
Subject: Re: Solr Cloud reclaiming disk space from deleted documents


To prevent it from re occurring you could monitor index size and once above
a
certain size threshold add another machine and split the shard between
existing
and new machine.
On Apr 20, 2015 9:10 PM, "Rishi Easwaran"
<rishi.easwaran@aol.com> wrote:

> So is there anything that can be done from
a tuning perspective, to
> recover a shard that is 75%-90% full, other that get
rid of the index and
> rebuild the data?
>  Also to prevent this issue from
re-occurring, looks like we need make our
> system aggressive with segment
merges using lower merge factor
>
>
> Thanks,
> Rishi.
>
>
>
>
-----Original Message-----
> From: Shawn Heisey <apache@elyograg.org>
> To:
solr-user <solr-user@lucene.apache.org>
> Sent: Mon, Apr 20, 2015 11:25 am
>
Subject: Re: Solr Cloud reclaiming disk space from deleted documents
>
>
> On
4/20/2015 8:44 AM, Rishi Easwaran wrote:
> > Yeah I noticed that. Looks like
>
optimize won't work since on some disks we are already pretty full.
> > Any
>
thoughts on increasing/decreasing <mergeFactor>10</mergeFactor>  or
>
ConcurrentMergeScheduler to make solr do merges faster.
>
> You don't have to
do
> an optimize to need 2x disk space.  Even normal
> merging, if it happens
just
> right, can require the same disk space as a
> full optimize.  Normal
Solr
> operation requires that you have enough
> space for your index to reach
at least
> double size on occasion.
>
> Higher merge factors are better for
indexing speed,
> because merging
> happens less frequently.  Lower merge
factors are better for
> query
> speed, at least after the merging finishes,
because merging happens
> more
> frequently and there are fewer total segments
at any given moment.
>
> During a merge, there is so much I/O that query speed
is often
> negatively
> affected.
>
> Thanks,
> Shawn
>
>
>
>

 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message