lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject RE: Increased disk space usage 8.1.1 vs 7.7.1
Date Thu, 13 Jun 2019 15:34:02 GMT
Hello,

It has something to do with the skewed facet counts seen in another thread. To make a full
comparison i indexed the same set to a fresh 7.7 build. Without my DocValues error, there
is still a reasonable difference:

7.7 shard 1: 7.8 GB
7.7 shard 2: 7.3 GB

8.1 shard 1: 8.3 GB
8.1 shard 2: 5.9 GB

Strange enough, one is larger and the second a lot smaller, and overall 8.1 takes about 1
GB less.

So it was my DocValues error that caused 8.1 locally to be larger than the old 7.7 production.

My bad, again!

Many thanks,
Markus 
 
-----Original message-----
> From:Shawn Heisey <apache@elyograg.org>
> Sent: Thursday 13th June 2019 13:42
> To: solr-user@lucene.apache.org
> Subject: Re: Increased disk space usage 8.1.1 vs 7.7.1
> 
> On 6/13/2019 4:19 AM, Markus Jelsma wrote:
> > We are upgrading to Solr 8. One of our reindexed collections takes a GB more than
the production uses which is on 7.7.1. Production also has deleted documents. This means Solr
8 somehow uses more disk space. I have checked both Solr and Lucene's CHANGES but no ticket
was immediately obvious.
> 
> Did you index to a core with nothing in it, or reindex on an existing 
> index without deleting everything first and letting Lucene erase all the 
> segments?
> 
> If you reindexed into an existing index, you could simply have deleted 
> documents taking up the extra space.  Full comparison would need to be 
> done after optimizing both indexes to clear out deleted documents.
> 
> You're probably already aware that optimizing in production is 
> discouraged, unless you're willing to do it frequently ... which gets 
> expensive with large indexes.
> 
> If the size is 1GB larger after both indexes are optimized to clear 
> deleted documents, then the other replies you've gotten will be important.
> 
> Thanks,
> Shawn
> 

Mime
View raw message