lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar R. Aiyengar" <andyetitmo...@gmail.com>
Subject Re: Whole RAM consumed while Indexing.
Date Mon, 16 Mar 2015 20:59:18 GMT
Yes, and doing so is painful and takes lots of people and hardware
resources to get there for large amounts of data and queries :)

As Erick says, work backwards from 60s and first establish how high the
commit interval can be to satisfy your use case..
On 16 Mar 2015 16:04, "Erick Erickson" <erickerickson@gmail.com> wrote:

> First start by lengthening your soft and hard commit intervals
> substantially. Start with 60000 and work backwards I'd say.
>
> Ramkumar has tuned the heck out of his installation to get the commit
> intervals to be that short ;).
>
> I'm betting that you'll see your RAM usage go way down, but that' s a
> guess until you test.
>
> Best,
> Erick
>
> On Sun, Mar 15, 2015 at 10:56 PM, Nitin Solanki <nitinmlvya@gmail.com>
> wrote:
> > Hi Erick,
> >             You are saying correct. Something, **"overlapping searchers"
> > warning messages** are coming in logs.
> > **numDocs numbers** are changing when documents are adding at the time of
> > indexing.
> > Any help?
> >
> > On Sat, Mar 14, 2015 at 11:24 PM, Erick Erickson <
> erickerickson@gmail.com>
> > wrote:
> >
> >> First, the soft commit interval is very short. Very, very, very, very
> >> short. 300ms is
> >> just short of insane unless it's a typo ;).
> >>
> >> Here's a long background:
> >>
> >>
> https://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >>
> >> But the short form is that you're opening searchers every 300 ms. The
> >> hard commit is better,
> >> but every 3 seconds is still far too short IMO. I'd start with soft
> >> commits of 60000 and hard
> >> commits of 60000 (60 seconds), meaning that you're going to have to
> >> wait 1 minute for
> >> docs to show up unless you explicitly commit.
> >>
> >> You're throwing away all the caches configured in solrconfig.xml more
> >> than 3 times a second,
> >> executing autowarming, etc, etc, etc....
> >>
> >> Changing these to longer intervals might cure the problem, but if not
> >> then, as Hoss would
> >> say, "details matter". I suspect you're also seeing "overlapping
> >> searchers" warning messages
> >> in your log, and it;s _possible_ that what's happening is that you're
> >> just exceeding the
> >> max warming searchers and never opening a new searcher with the
> >> newly-indexed documents.
> >> But that's a total shot in the dark.
> >>
> >> How are you looking for docs (and not finding them)? Does the numDocs
> >> number in
> >> the solr admin screen change?
> >>
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Mar 12, 2015 at 10:27 PM, Nitin Solanki <nitinmlvya@gmail.com>
> >> wrote:
> >> > Hi Alexandre,
> >> >
> >> >
> >> > *Hard Commit* is :
> >> >
> >> >      <autoCommit>
> >> >        <maxTime>${solr.autoCommit.maxTime:3000}</maxTime>
> >> >        <openSearcher>false</openSearcher>
> >> >      </autoCommit>
> >> >
> >> > *Soft Commit* is :
> >> >
> >> > <autoSoftCommit>
> >> >     <maxTime>${solr.autoSoftCommit.maxTime:300}</maxTime>
> >> > </autoSoftCommit>
> >> >
> >> > And I am committing 20000 documents each time.
> >> > Is it good config for committing?
> >> > Or I am good something wrong ?
> >> >
> >> >
> >> > On Fri, Mar 13, 2015 at 8:52 AM, Alexandre Rafalovitch <
> >> arafalov@gmail.com>
> >> > wrote:
> >> >
> >> >> What's your commit strategy? Explicit commits? Soft commits/hard
> >> >> commits (in solrconfig.xml)?
> >> >>
> >> >> Regards,
> >> >>    Alex.
> >> >> ----
> >> >> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> >> >> http://www.solr-start.com/
> >> >>
> >> >>
> >> >> On 12 March 2015 at 23:19, Nitin Solanki <nitinmlvya@gmail.com>
> wrote:
> >> >> > Hello,
> >> >> >           I have written a python script to do 20000 documents
> >> indexing
> >> >> > each time on Solr. I have 28 GB RAM with 8 CPU.
> >> >> > When I started indexing, at that time 15 GB RAM was freed. While
> >> >> indexing,
> >> >> > all RAM is consumed but **not** a single document is indexed.
Why
> so?
> >> >> > And it through *HTTPError: HTTP Error 503: Service Unavailable*
in
> >> python
> >> >> > script.
> >> >> > I think it is due to heavy load on Zookeeper by which all nodes
> went
> >> >> down.
> >> >> > I am not sure about that. Any help please..
> >> >> > Or anything else is happening..
> >> >> > And how to overcome this issue.
> >> >> > Please assist me towards right path.
> >> >> > Thanks..
> >> >> >
> >> >> > Warm Regards,
> >> >> > Nitin Solanki
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message