lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elodie Sannier <elodie.sann...@kelkoo.fr>
Subject Re: SolrIndexSearcher accumulation
Date Wed, 19 Apr 2017 16:00:12 GMT
Yes, I didn't copy all our code but we also do extraReq.close(); in a
finally block. It was not the problem.

On 04/19/2017 11:53 AM, Mikhail Khludnev wrote:
> If you create SolrQueryRequest make sure you close it then, since it's
> necessary to release a searcher.
>
> On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier <elodie.sannier@kelkoo.fr>
> wrote:
>
>> Hello,
>>
>> We have found how to fix the problem.
>> When we update the original SolrQueryResponse object, we need to create
>> a new BasicResultContext object with the extra response.
>>
>> Simplified code :
>>
>> public class CustomSearchHandler extends SearchHandler {
>>
>> public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
>> rsp) throws Exception {
>>
>>    SolrQueryRequest extraReq = createExtraRequest();
>>    SolrQueryResponse extraRsp = new SolrQueryResponse();
>>
>>    super.handleRequestBody(extraReq, extraRsp);
>>
>>    ResultContext extraRc = (ResultContext) extraRsp.getResponse();
>>
>>    // code with memory leak !!
>>    rsp.addResponse(extraRc);
>>
>>    // code without memory leak
>>    ResultContext extraRcClone = new BasicResultContext(extraRc.get
>> DocList(),
>>              rsp.getReturnFields(), req.getSearcher(),
>> extraRc.getQuery(), req);
>>    rsp.addResponse(extraRcClone);
>>
>> }
>>
>> }
>>
>> We don't know why we need to create a new BasicResultContext to properly
>> manage searchers. Do you know why ?
>>
>> Elodie
>>
>>
>> On 04/07/2017 04:14 PM, Rick Leir wrote:
>>
>>> Hi Gerald
>>> The best solution in my mind is to look at the custom code and try to
>>> find a way to remove it from your system. Solr queries can be complex, and
>>> I hope there is a way to get the results you need. Would you like to say
>>> what results you want to get, and what Solr queries you have tried?
>>> I realize that in large organizations it is difficult to suggest change.
>>> Cheers -- Rick
>>>
>>> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <apache@elyograg.org>
>>> wrote:
>>>
>>>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>>
>>>>>      We have some custom code that extends SearchHandler to be able to
>>>>>
>>>> :
>>>>
>>>>>           - do an extra request
>>>>>           - merge/combine the original request and the extra request
>>>>> results
>>>>>
>>>>>      On Solr 5.x, our code was working very well, now with Solr 6.x we
>>>>> have the following issue:  the number of SolrIndexSearcher are
>>>>> increasing (we can see them in the admin view > Plugins/ Stats >
Core
>>>>>
>>>> ).
>>>>
>>>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>>>          - the memory used by Solr is increasing => OOM after a long
>>>>> period of time in production
>>>>>          - some files in the index has been deleted from the system but
>>>>> the Solr JVM still hold them => ("fake") Full disk after a long
>>>>>
>>>> period
>>>>
>>>>> of time in production
>>>>>
>>>>>      We are wondering,
>>>>>         - what has changed between Solr 5.x and Solr 6.x in the
>>>>> management of the SolrIndexSearcher ?
>>>>>         - what would be the best way, in a Solr plugin, to perform 2
>>>>> queries and merge the results to a single SolrQueryResponse ?
>>>>>
>>>> I hesitated to send a reply because when it comes right down to it, I
>>>> do
>>>> not know a whole lot about deep Solr internals.  I tend to do my work
>>>> with the code at a higher level, and don't dive down in the depths all
>>>> that often.  I am slowly learning, though.  You may need to wait for a
>>>> reply from someone who really knows those internals.
>>>>
>>>> It looks like you and I participated in a discussion last month where
>>>> you were facing a similar problem with searchers -- deleted index files
>>>> being held open.  How did that turn out?  Seems like if that problem
>>>> were solved, it would also solve this problem.
>>>>
>>>> Very likely, the fact that the plugin worked correctly in 5.x was
>>>> actually a bug in Solr related to reference counting, one that has been
>>>> fixed in later versions.
>>>>
>>>> You may need to use a paste website or a file-sharing website to share
>>>> all your plugin code so that people can get a look at it.  The list has
>>>> a habit of deleting attachments.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>> Kelkoo SAS
>> Société par Actions Simplifiée
>> Au capital de € 4.168.964,30
>> Siège social : 158 Ter Rue du Temple 75003 Paris
>> 425 093 069 RCS Paris
>>
>> Ce message et les pièces jointes sont confidentiels et établis à
>> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
>> destinataire de ce message, merci de le détruire et d'en avertir
>> l'expéditeur.
>>
>
>


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive
de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire
et d'en avertir l'expéditeur.

Mime
View raw message