lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Hurt <bhur...@gmail.com>
Subject Re: Closing the IndexSearcher/IndexWriter for a core
Date Mon, 03 Aug 2015 23:11:32 GMT
Some further information:

The main things use memory that I see from my heap dump are:

1. Arrays of org.apache.lucene.util.fst.FST$Arc classes- which mainly seem
to hold nulls.  The ones of these I've investigated have been held by
org.apache.lucene.util.fst.FST objects, I have 38 cores open and have over
121,000 of these arrays, taking up over 126M of space.

2. Byte arrays, of which I have 384,000 of, taking up 106M of space.

When I trace the cycle of references up, I've always ended up at an
IndexSearcher or IndexWriter class, causing me to assume the problem was
that I was simply opening up too many cores, but I could be mistaken.

This was on a freshly started system without many cores having been touched
yet- so the memory usage, while larger than I expect, isn't critical yet.
It does become critical as the number of cores increases.

Thanks,

Brian



On Mon, Aug 3, 2015 at 4:57 PM, Brian Hurt <bhurt42@gmail.com> wrote:

>
> Is there are an easy way for a client to tell Solr to close or release the
> IndexSearcher and/or IndexWriter for a core?
>
> I have a use case where we're creating a lot of cores with not that many
> documents per zone (a few hundred to maybe 10's of thousands).  Writes come
> in batches, and reads also tend to be bursty, if less so than the writes.
>
> And we're having problems with ram usage on the server.  Poking around a
> heap dump, the problem is that every IndexSearcher or IndexWriter being
> opened is taking up large amounts of memory.
>
> I've looked at the unload call, and while it is unclear, it seems like it
> deletes the data on disk as well.  I don't want to delete the data on disk,
> I just want to unload the searcher and writer, and free up the memory.
>
> So I'm wondering if there is a call I can make when I know or suspect that
> the core isn't going to be used in the near future to release these objects
> and return the memory?  Or a configuration option I can set to do so after,
> say, being idle for 5 seconds?  It's OK for there to be a performance hit
> the first time I reopen the core.
>
> Thanks,
>
> Brian
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message