lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <m...@apache.org>
Subject Re: SolrIndexSearcher accumulation
Date Wed, 19 Apr 2017 09:53:44 GMT
If you create SolrQueryRequest make sure you close it then, since it's
necessary to release a searcher.

On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier <elodie.sannier@kelkoo.fr>
wrote:

> Hello,
>
> We have found how to fix the problem.
> When we update the original SolrQueryResponse object, we need to create
> a new BasicResultContext object with the extra response.
>
> Simplified code :
>
> public class CustomSearchHandler extends SearchHandler {
>
> public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
> rsp) throws Exception {
>
>   SolrQueryRequest extraReq = createExtraRequest();
>   SolrQueryResponse extraRsp = new SolrQueryResponse();
>
>   super.handleRequestBody(extraReq, extraRsp);
>
>   ResultContext extraRc = (ResultContext) extraRsp.getResponse();
>
>   // code with memory leak !!
>   rsp.addResponse(extraRc);
>
>   // code without memory leak
>   ResultContext extraRcClone = new BasicResultContext(extraRc.get
> DocList(),
>             rsp.getReturnFields(), req.getSearcher(),
> extraRc.getQuery(), req);
>   rsp.addResponse(extraRcClone);
>
> }
>
> }
>
> We don't know why we need to create a new BasicResultContext to properly
> manage searchers. Do you know why ?
>
> Elodie
>
>
> On 04/07/2017 04:14 PM, Rick Leir wrote:
>
>> Hi Gerald
>> The best solution in my mind is to look at the custom code and try to
>> find a way to remove it from your system. Solr queries can be complex, and
>> I hope there is a way to get the results you need. Would you like to say
>> what results you want to get, and what Solr queries you have tried?
>> I realize that in large organizations it is difficult to suggest change.
>> Cheers -- Rick
>>
>> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey <apache@elyograg.org>
>> wrote:
>>
>>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>
>>>>     We have some custom code that extends SearchHandler to be able to
>>>>
>>> :
>>>
>>>>          - do an extra request
>>>>          - merge/combine the original request and the extra request
>>>> results
>>>>
>>>>     On Solr 5.x, our code was working very well, now with Solr 6.x we
>>>> have the following issue:  the number of SolrIndexSearcher are
>>>> increasing (we can see them in the admin view > Plugins/ Stats > Core
>>>>
>>> ).
>>>
>>>> As SolrIndexSearcher are accumulating, we have the following issues :
>>>>         - the memory used by Solr is increasing => OOM after a long
>>>> period of time in production
>>>>         - some files in the index has been deleted from the system but
>>>> the Solr JVM still hold them => ("fake") Full disk after a long
>>>>
>>> period
>>>
>>>> of time in production
>>>>
>>>>     We are wondering,
>>>>        - what has changed between Solr 5.x and Solr 6.x in the
>>>> management of the SolrIndexSearcher ?
>>>>        - what would be the best way, in a Solr plugin, to perform 2
>>>> queries and merge the results to a single SolrQueryResponse ?
>>>>
>>> I hesitated to send a reply because when it comes right down to it, I
>>> do
>>> not know a whole lot about deep Solr internals.  I tend to do my work
>>> with the code at a higher level, and don't dive down in the depths all
>>> that often.  I am slowly learning, though.  You may need to wait for a
>>> reply from someone who really knows those internals.
>>>
>>> It looks like you and I participated in a discussion last month where
>>> you were facing a similar problem with searchers -- deleted index files
>>> being held open.  How did that turn out?  Seems like if that problem
>>> were solved, it would also solve this problem.
>>>
>>> Very likely, the fact that the plugin worked correctly in 5.x was
>>> actually a bug in Solr related to reference counting, one that has been
>>> fixed in later versions.
>>>
>>> You may need to use a paste website or a file-sharing website to share
>>> all your plugin code so that people can get a look at it.  The list has
>>> a habit of deleting attachments.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>



-- 
Sincerely yours
Mikhail Khludnev

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message