lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: index format error because disk full - possible Lucene bug?
Date Fri, 09 Jul 2010 11:59:06 GMT
I believe I've seen a similar condition a few times.
A segments file referring zero-length segment files after a disk full event.

On Fri, Jul 9, 2010 at 13:37, Michael McCandless
<lucene@mikemccandless.com> wrote:
> I responded on the original thread.
>
> Disk full should never cause index corruption except on very old
> versions of Lucene...
>
> Mike
>
> On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <goksron@gmail.com> wrote:
>> I think he saying that there is a race condition involving writing out
>> the buffered changes from ram, doing deletes, doing commits and
>> writing out new segment files when Lucene runs out of disk. This race
>> condition (or possibly a swallowed I/O Exception) may cause bogus
>> segment files, and the index becomes unuseable even though the
>> existing files make up a clean index.
>>
>> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <goksron@gmail.com> wrote:
>>> Forwarded to lucene-dev as a possible bug.
>>>
>>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <fancyerii@gmail.com> wrote:
>>>> I use SegmentInfos to read the segment_N file and found the error is
>>>> that it try to load deletedDocs but the .del file's size is 0(because
>>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>>>> deleted Docs.
>>>> But I think there is some bug. The logic of  write my be -- it first
>>>> writes the .del file then write the segment_N file. But it only write
>>>> to buffer and don't flush to disk immediately. So when disk full. it
>>>> may happen that segment_N file is flushed but del file faild.
>>>>
>>>> 2010/7/8 Lance Norskog <goksron@gmail.com>:
>>>>> If autocommit does not to an automatic rollback, that is a serious bug.
>>>>>
>>>>> There should be a way to detect that an automatic rollback has
>>>>> happened, but I don't know what it is. Maybe something in the Solr
>>>>> MBeans?
>>>>>
>>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <ken.foster@realestate.com>
wrote:
>>>>>>
>>>>>> I haven't used this myself, but Solr supports a
>>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback
>>>>>> function. It is supposed to rollback to the state at the previous
commit. So
>>>>>> you may want to turn off auto-commit on the index you are updating
if you
>>>>>> want to control what that last commit level is.
>>>>>>
>>>>>> However, in your case if the index gets corrupted due to a disk full
>>>>>> situation, I don't know what rollback would do, if anything, to help.
You
>>>>>> may need to play with the scenario to see what would happen.
>>>>>>
>>>>>> If you are using the DataImportHandler it may handle the rollback
for
>>>>>> you...again, however, it may not deal with disk full situations gracefully
>>>>>> either.
>>>>>> --
>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> goksron@gmail.com
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message