lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr leaking references to deleted files
Date Tue, 31 May 2016 20:26:25 GMT
Cool, please let us know what you find out.

On Tue, May 31, 2016 at 8:34 AM, Gavin Harcourt
<gharcourt@brainboxdigital.com> wrote:
> Those two bugs would make sense as we have been reloading the cores quite
> frequently recently to apply new config and schema changes. I'll keep an eye
> on the situation now our reload spree has ended and see if it recurs.
>
> Thanks,
> Gavin.
>
> On 31/05/16 16:14, Erick Erickson wrote:
>>
>> Possibly:
>> SOLR-9116 or SOLR-9117? Note those two require that the core be
>> reloaded, so you have to be doing something a bit unusual for them to
>> be the problem.
>>
>> Best,
>> Erick
>>
>> On Tue, May 31, 2016 at 5:41 AM, Gavin Harcourt
>> <gharcourt@brainboxdigital.com> wrote:
>>>
>>> Hi All,
>>>
>>> I've noticed on some of my solr nodes that the disk usage is increasing
>>> over
>>> time. After checking the output of lsof I found hundreds of references to
>>> deleted index files being held by solr. This totaled 24GB on a 16GB
>>> index. A
>>> restart of solr can obviously fix this but this is not an ideal solution.
>>> We
>>> are running solr 5.4.0 on OpenJDK  1.8.0_91. We are using the Concurrent
>>> Mark Sweep GC although I've also seen the same problem on nodes using the
>>> G1
>>> GC. Our update handler has autoCommit and softAutoCommit enabled (at
>>> different intervals). We are using solr cloud and have multiple shards
>>> with
>>> 2 nodes each in our collections. I've not seen any pattern between this
>>> appearing on leaders or replicas. Not all my nodes appear to be
>>> exhibiting
>>> the problem either. Our usage pattern does involve a lot of churn in our
>>> index with the majority of documents being updated/deleted every day.
>>>
>>> Searching JIRA and the web in general I could only find references to
>>> this
>>> sort of problem when running solr in tomcat. Can anyone suggest a reason
>>> why
>>> this might be happening or a way I can manage it without needing to
>>> restart
>>> solr?
>>>
>>> Example lsof output:
>>> java       1100      s123  DEL       REG 202,3               8919406
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt
>>> java       1100      s123  DEL       REG 202,3               8919159
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd
>>> java       1100      s123  DEL       REG 202,3               8919150
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim
>>> java       1100      s123  DEL       REG 202,3               8919094
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim
>>> java       1100      s123  DEL       REG 202,3               8919103
>>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd
>>>
>>> Regards,
>>> Gavin.
>>>
>

Mime
View raw message