From dev-return-201378-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org Wed Apr 1 22:19:04 2015 Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 59EE317A53 for ; Wed, 1 Apr 2015 22:19:04 +0000 (UTC) Received: (qmail 73565 invoked by uid 500); 1 Apr 2015 22:18:54 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 73504 invoked by uid 500); 1 Apr 2015 22:18:53 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 73493 invoked by uid 99); 1 Apr 2015 22:18:53 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2015 22:18:53 +0000 Date: Wed, 1 Apr 2015 22:18:53 +0000 (UTC) From: "Michael McCandless (JIRA)" To: dev@lucene.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (LUCENE-5941) IndexWriter.forceMerge documentation error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/LUCENE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-5941. ---------------------------------------- Resolution: Fixed I'll leave this issue unchanged (issues once released should be write once), and fix this under LUCENE-6386. > IndexWriter.forceMerge documentation error > ------------------------------------------ > > Key: LUCENE-5941 > URL: https://issues.apache.org/jira/browse/LUCENE-5941 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index > Reporter: Shai Erera > Assignee: Shai Erera > Fix For: Trunk, 5.0 > > Attachments: LUCENE-5941.patch, LUCENE-5941.patch, LUCENE-5941.patch > > > IndexWriter.forceMerge documents that it requires up to 3X *FREE* space in order to run successfully. We even go further with it and test it in TestIWForceMerge.testForceMergeTempSpaceUsage(). But I think that's wrong. I cannot think of a situation where we consume 3X *additional* space during merge: > * 1X - that's the source segments to be merged > * 2X - that's the result non-CFS merged segment > * 3X - that's the CFS creation > At no point do we publish the non-CFS merged segment, therefore the merge, as I understand it, only consumes up to 2X additional space during that merge. > And anyway, we only require 2X of additional space of the *largest* merge (or total batch of running merges, depends on your MergeScheduler), not the whole index size. This is an important observation, since if you e.g. have a 500GB index, users shouldn't think they need to reserve an additional 1TB for merging, since most of their big segments won't be merged by default anyway (TieredMP defaults to 5GB largest segment). > I'll post a patch which fixes the documentation and the test. If anyone can think of a scenario where we consume up to 3X *additional* space, please chime, and I'll only modify IW.forceMerge documentation to explain that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org