lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <abenede...@apache.org>
Subject Re: Solr hard commit
Date Wed, 04 Nov 2015 18:07:27 GMT
Just thinking loud, but Mapping is a OS feature.
For example , if a recovery is happening ( and it involves less than 100
docs of difference), the Tlog will be read to reproduce the data.
This will cause the Tlog to be accessed and ideally memory mapped.
Am I correct ?

Cheers

On 28 October 2015 at 01:56, Erick Erickson <erickerickson@gmail.com> wrote:

> yep. Although I won't guarantee that the tlog won't be MMapped
> even if this is turned off.
>
> On Tue, Oct 27, 2015 at 4:21 PM, Rallavagu <rallavagu@gmail.com> wrote:
> > Is it related to this config?
> >
> > <!-- If true, IndexReaders will be opened/reopened from the IndexWriter
> >          instead of from the Directory. Hosts in a master/slave setup
> >          should have this set to false while those in a SolrCloud
> >          cluster need to be set to true. Default: true
> >       -->
> >     <!--
> >     <nrtMode>true</nrtMode>
> >
> > this will be true by default.
> >
> > On 10/27/15 1:24 PM, Erick Erickson wrote:
> >>
> >> Hmm, the tlog. AFAIK, that's because of
> >> "real time get" (Yonik/Mark/Whoever, please
> >> correct if wrong).
> >>
> >> This is a feature where when fetching a document as
> >> the result of a search it insures that you get the most
> >> recent version whether it's been committed or not.....
> >>
> >> The tlog isn't relevant for searching however.
> >>
> >> Best,
> >> Erick
> >>
> >> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <rallavagu@gmail.com> wrote:
> >>>
> >>>
> >>>
> >>> On 10/27/15 8:43 AM, Erick Erickson wrote:
> >>>>
> >>>>
> >>>> bq: So, the updated file(s) on the disk automatically read into memory
> >>>> as they are Memory mapped?
> >>>
> >>>
> >>> Yes.
> >>>>
> >>>>
> >>>>
> >>>> Not quite sure why you care, curiosity or is there something you're
> >>>> trying to accomplish?
> >>>
> >>>
> >>> This is out of curiosity. So, I can get better understanding of Solr's
> >>> memory usage (heap & mmap).
> >>>
> >>>>
> >>>> The contents of the index's segment files are read into virtual memory
> >>>> by MMapDirectory as needed to satisfy queries. Which is the point of
> >>>> autowarming BTW.
> >>>
> >>>
> >>>
> >>> Ok. But, I have noticed that even "tlog" files are memory mapped
> (output
> >>> from "lsof") in addition to all other files under "data" directory.
> >>>
> >>>>
> >>>> commit in the following is either hard commit with openSearcher=true
> >>>> or soft commit.
> >>>
> >>>
> >>>
> >>> Hard commit is setup with openSearcher=false and softCommit is setup
> for
> >>> every 2 min.
> >>>
> >>>>
> >>>> Segments that have been created (closed actually) after the last
> >>>> commit  are _not_ read at all until the next searcher is opened via
> >>>> another commit. Nothing is done with these new segments before the new
> >>>> searcher is opened which you control with your commit strategy.
> >>>
> >>>
> >>>
> >>> I see. Thanks for the insight.
> >>>
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <rallavagu@gmail.com>
> wrote:
> >>>>>
> >>>>>
> >>>>> Erick, Thanks for clarification. I was under impression that
> >>>>> MMapDirectory
> >>>>> is being used for both read/write operations. Now, I see how it
is
> >>>>> being
> >>>>> used. Essentially, it only reads from MMapDirectory and writes
> directly
> >>>>> to
> >>>>> disk. So, the updated file(s) on the disk automatically read into
> >>>>> memory
> >>>>> as
> >>>>> they are Memory mapped?
> >>>>>
> >>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> You're really looking at this backwards. The MMapDirectory stuff
is
> >>>>>> for Solr (Lucene, really) _reading_ data from closed segment
files.
> >>>>>>
> >>>>>> When indexing, there are internal memory structures that are
flushed
> >>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
> >>>>>>
> >>>>>> So the question is really moot ;)
> >>>>>>
> >>>>>> Best,
> >>>>>> Erick
> >>>>>>
> >>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <rallavagu@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> All,
> >>>>>>>
> >>>>>>> Are memory mapped files (mmap) flushed to disk during "hard
> commit"?
> >>>>>>> If
> >>>>>>> yes,
> >>>>>>> should we disable OS level (Linux for example) memory mapped
flush?
> >>>>>>>
> >>>>>>> I am referring to following for mmap files for Lucene/Solr
> >>>>>>>
> >>>>>>>
> >>>>>>>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> >>>>>>>
> >>>>>>> Linux level flush
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
> >>>>>>>
> >>>>>>> Solr's hard and soft commit
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >>>>>>>
> >>>>>>> Thanks in advance.
>



-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message