lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: index format error because disk full - possible Lucene bug?
Date Fri, 09 Jul 2010 18:51:00 GMT
Which version of Lucene, and which OS/IO system?

Is it possible fsync lies in your env?

Mike

On Fri, Jul 9, 2010 at 7:59 AM, Earwin Burrfoot <earwin@gmail.com> wrote:
> I believe I've seen a similar condition a few times.
> A segments file referring zero-length segment files after a disk full event.
>
> On Fri, Jul 9, 2010 at 13:37, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> I responded on the original thread.
>>
>> Disk full should never cause index corruption except on very old
>> versions of Lucene...
>>
>> Mike
>>
>> On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <goksron@gmail.com> wrote:
>>> I think he saying that there is a race condition involving writing out
>>> the buffered changes from ram, doing deletes, doing commits and
>>> writing out new segment files when Lucene runs out of disk. This race
>>> condition (or possibly a swallowed I/O Exception) may cause bogus
>>> segment files, and the index becomes unuseable even though the
>>> existing files make up a clean index.
>>>
>>> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <goksron@gmail.com> wrote:
>>>> Forwarded to lucene-dev as a possible bug.
>>>>
>>>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <fancyerii@gmail.com> wrote:
>>>>> I use SegmentInfos to read the segment_N file and found the error is
>>>>> that it try to load deletedDocs but the .del file's size is 0(because
>>>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore
>>>>> deleted Docs.
>>>>> But I think there is some bug. The logic of  write my be -- it first
>>>>> writes the .del file then write the segment_N file. But it only write
>>>>> to buffer and don't flush to disk immediately. So when disk full. it
>>>>> may happen that segment_N file is flushed but del file faild.
>>>>>
>>>>> 2010/7/8 Lance Norskog <goksron@gmail.com>:
>>>>>> If autocommit does not to an automatic rollback, that is a serious
bug.
>>>>>>
>>>>>> There should be a way to detect that an automatic rollback has
>>>>>> happened, but I don't know what it is. Maybe something in the Solr
>>>>>> MBeans?
>>>>>>
>>>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <ken.foster@realestate.com>
wrote:
>>>>>>>
>>>>>>> I haven't used this myself, but Solr supports a
>>>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
rollback
>>>>>>> function. It is supposed to rollback to the state at the previous
commit. So
>>>>>>> you may want to turn off auto-commit on the index you are updating
if you
>>>>>>> want to control what that last commit level is.
>>>>>>>
>>>>>>> However, in your case if the index gets corrupted due to a disk
full
>>>>>>> situation, I don't know what rollback would do, if anything,
to help. You
>>>>>>> may need to play with the scenario to see what would happen.
>>>>>>>
>>>>>>> If you are using the DataImportHandler it may handle the rollback
for
>>>>>>> you...again, however, it may not deal with disk full situations
gracefully
>>>>>>> either.
>>>>>>> --
>>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html
>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lance Norskog
>>>>>> goksron@gmail.com
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lance Norskog
>>>> goksron@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
>
>
> --
> Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
> Phone: +7 (495) 683-567-4
> ICQ: 104465785
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message