lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elodie Sannier <elodie.sann...@kelkoo.fr>
Subject Re: SolrIndexSearcher accumulation
Date Wed, 19 Apr 2017 09:35:46 GMT
Hello,

We have found how to fix the problem.
When we update the original SolrQueryResponse object, we need to create
a new BasicResultContext object with the extra response.

Simplified code :

public class CustomSearchHandler extends SearchHandler {

public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
rsp) throws Exception {

   SolrQueryRequest extraReq = createExtraRequest();
   SolrQueryResponse extraRsp = new SolrQueryResponse();

   super.handleRequestBody(extraReq, extraRsp);

   ResultContext extraRc = (ResultContext) extraRsp.getResponse();

   // code with memory leak !!
   rsp.addResponse(extraRc);

   // code without memory leak
   ResultContext extraRcClone = new BasicResultContext(extraRc.getDocList(),
             rsp.getReturnFields(), req.getSearcher(),
extraRc.getQuery(), req);
   rsp.addResponse(extraRcClone);

}

}

We don't know why we need to create a new BasicResultContext to properly
manage searchers. Do you know why ?

Elodie

On 04/07/2017 04:14 PM, Rick Leir wrote:
> Hi Gerald
> The best solution in my mind is to look at the custom code and try to find a way to remove
it from your system. Solr queries can be complex, and I hope there is a way to get the results
you need. Would you like to say what results you want to get, and what Solr queries you have
tried?
> I realize that in large organizations it is difficult to suggest change.
> Cheers -- Rick
>
> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <apache@elyograg.org> wrote:
>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>     We have some custom code that extends SearchHandler to be able to
>> :
>>>          - do an extra request
>>>          - merge/combine the original request and the extra request
>>> results
>>>
>>>     On Solr 5.x, our code was working very well, now with Solr 6.x we
>>> have the following issue:  the number of SolrIndexSearcher are
>>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>> ).
>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>         - the memory used by Solr is increasing => OOM after a long
>>> period of time in production
>>>         - some files in the index has been deleted from the system but
>>> the Solr JVM still hold them => ("fake") Full disk after a long
>> period
>>> of time in production
>>>
>>>     We are wondering,
>>>        - what has changed between Solr 5.x and Solr 6.x in the
>>> management of the SolrIndexSearcher ?
>>>        - what would be the best way, in a Solr plugin, to perform 2
>>> queries and merge the results to a single SolrQueryResponse ?
>> I hesitated to send a reply because when it comes right down to it, I
>> do
>> not know a whole lot about deep Solr internals.  I tend to do my work
>> with the code at a higher level, and don't dive down in the depths all
>> that often.  I am slowly learning, though.  You may need to wait for a
>> reply from someone who really knows those internals.
>>
>> It looks like you and I participated in a discussion last month where
>> you were facing a similar problem with searchers -- deleted index files
>> being held open.  How did that turn out?  Seems like if that problem
>> were solved, it would also solve this problem.
>>
>> Very likely, the fact that the plugin worked correctly in 5.x was
>> actually a bug in Solr related to reference counting, one that has been
>> fixed in later versions.
>>
>> You may need to use a paste website or a file-sharing website to share
>> all your plugin code so that people can get a look at it.  The list has
>> a habit of deleting attachments.
>>
>> Thanks,
>> Shawn


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive
de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire
et d'en avertir l'expéditeur.

Mime
View raw message