lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Nithian <>
Subject Re: How real-time are Solr/Lucene queries?
Date Wed, 26 May 2010 06:24:24 GMT
This is an interesting discussion and I have a few questions:
1) My apologies but I haven't been following the NRT patch beyond what was
presented at a meetup some months back and the wiki but what is the status
of it in Solr?
2) What are typical/accepted definitions of "Real Time" vs "Near Real Time"?
==> Related to Grant's points earlier.
3) I could understand POSTing a document to a server and then turning around
and searching for it on the same server but what about a replicated
environment and how do you prevent caches from being blown and constantly
re-warmed (hence performance degradation)? I could set Solr replication once
per minute or less but then the caches caches would be regenerated each
minute which I suspect is not cheap.

Note: I am curious about this from a typical web-based application
perspective as opposed to an embedded/desktop application like context
described earlier in this thread.


On Tue, May 25, 2010 at 10:28 AM, Thomas J. Buhr

> My documents are all quite small if not down right tiny, there is not much
> analysis to do. I plan to mainly use Solr for indexing application
> configuration data which there is a lot of and I have all pre-formated.
> Since it is a music application there are many score templates, scale and
> rhythm strings, notation symbol skins, etc. Then there are slightly more
> usual things to index like application help pages and tutorials.
> In terms of queries per second there will be a lot being fired by our
> painter. In our application data is flowing into a painter who in turn
> delegates specific painting tasks to renderer objects. These renderer
> objects then make many queries extremely fast to the embedded Solr indexes
> for data they need, such as layout and style values.
> Believe me there is a lot of detailed data involved in music notation and
> abstracting it into configurations in the form of index documents is a good
> way to manage such data. Further, the data in the form of documents work as
> a form of plugins so that alternate configurations for different notation
> types can be added to the index. Then via simple search it is possible to
> dialup a certain set of documents which contain all the details of a given
> notation. Mean while the renderer objects remain generic and are just
> reconfigured with the different indexed configuration documents.
> Will making many fast queries from renderers to an embedded local Solr
> index slow my painting down?
> Thom
> On 2010-05-25, at 6:09 AM, Grant Ingersoll wrote:
> > How many docs are in the batch you are pulling down?  How many
> docs/second do you expect on the index size?  How big are the docs?  What do
> you expect in terms of queries per second?  How fast do new documents need
> to be available on the local server?  How much analysis do you have to do?
>  Also, define Real Time.  You'd be surprised at the number of people I talk
> to who think they need Real Time, but then when you ask them questions like
> I just did, they don't really need it.  I've seen Solr turn around new docs
> in as little as 30 seconds on commodity hardware w/o any special engineering
> effort and I've seen it faster than that with some engineering effort.  That
> isn't necessarily possible for every application, but...
> >
> > Despite the other suggestions, what you describe still looks feasible to
> me in Solr, pending the questions above (and some followups).
> >
> >
> > On May 21, 2010, at 4:05 AM, Thomas J. Buhr wrote:
> >
> >> Thanks for the new information. Its really great to see so many options
> for Lucene.
> >>
> >> In my scenario there are the following pieces:
> >>
> >> 1 - A local Java client with an embedded Solr instance and its own local
> index/s.
> >> 2 - A remote server running Solr with index/s that are more like a
> repository that local clients query for extra goodies.
> >> 3 - The client is also a JXTA node so it can share indexes or documents
> too.
> >> 4 - There is no browser involved what so ever.
> >>
> >> My music composing application is a local client that uses
> configurations which would become many different document types. A subset of
> these configurations will be bundled with the application and then many more
> would be made available via a server/s running Solr.
> >>
> >> I would not expect the queries which would be made from within the local
> client to be returned in real-time. I would only expect such queries to be
> made in reasonable time and returned to the client. The client would have
> its local Lucene index system (embedded Solr using SolrJ) which would be
> updated with the results of the query made to the Solr instance running on
> the remote server.
> >>
> >> Then the user on the client would issue queries to the local Lucene
> index/s to obtain results which are used to setup contexts for different
> aspects of the client. For example: an activated context for musical scales
> and rhythms used for creating musical notes, an activated context for
> rendering with layout and style information for different music symbol
> renderer types.
> >>
> >> I'm not yet sure but it may be best to make queries against the local
> Lucene index/s and then convert the results into some context objects, maybe
> an array or map (I'd like to learn more about how query results can be
> returned as arrays or maps as well). Then the tools and renderers which
> require the information in the contexts would do any real-time lookup
> directly from the context objects not the local or remote Lucene or Solr
> index/s. The local client is also a JXTA node so it can share its own
> index/s with fellow peers.
> >>
> >> This is how I envision this happening with my limited knowledge of
> Lucene/Solr at this time. What are your thoughts on the feasibility of such
> a scenario?
> >>
> >> I'm just reading through the Solr reference PDF now and looking over the
> Solr admin application. Looking at the Schema.xml it seems to be field not
> document oriented. From my point of view I think in terms of configuration
> types which would be documents. In the schema it seems like only fields are
> defined and it does not matter which configuration/document they belong to?
> I guess this is fine as long as the indexing takes into account my unique
> document types and I can search for them as a whole as well, not only for
> specific values across a set of indexed documents.
> >>
> >> Also, does the schema allow me to index certain documents into specific
> indexes or are they all just bunched together? I'd rather have unique
> indexes for specific document types. I've just read about multiple cores
> running under one Solr instance, is this the only way to support multiple
> indexes?
> >>
> >> I'm thinking of ordering the Lucene in Action v2 book which is due this
> month and also the Solr 1.4 book. Before I do I just need to understand a
> few things which is why I'm writing such a long message :-)
> >>
> >> Thom
> >>
> >>
> >> On 2010-05-21, at 2:12 AM, Ben Eliott wrote:
> >>
> >>> Further to earlier note re Lucandra.  I note that Cassandra, which
> Lucandra backs onto,  is 'eventually consistent',  so given your real-time
> requirements,  you may want to review this in the first instance, if
> Lucandra is of interest.
> >>>
> >>> On 21 May 2010, at 06:12, Walter Underwood wrote:
> >>>
> >>>> Solr is a very good engine, but it is not real-time. You can turn off
> the caches and reduce the delays, but it is fundamentally not real-time.
> >>>>
> >>>> I work at MarkLogic, and we have a real-time transactional search
> engine (and respository). If you are curious, contact me directly.
> >>>>
> >>>> I do like Solr for lots of applications -- I chose it when I was at
> Netflix.
> >>>>
> >>>> wunder
> >>>>
> >>>> On May 20, 2010, at 7:22 PM, Thomas J. Buhr wrote:
> >>>>
> >>>>> Hello Soir,
> >>>>>
> >>>>> Soir looks like an excellent API and its nice to have a tutorial
> makes it easy to discover the basics of what Soir does, I'm impressed. I can
> see plenty of potential uses of Soir/Lucene and I'm interested now in just
> how real-time the queries made to an index can be?
> >>>>>
> >>>>> For example, in my application I have time ordered data being
> processed by a paint method in real-time. Each piece of data is identified
> and its associated renderer is invoked. The Java2D renderer would then
> lookup any layout and style values it requires to render the current data it
> has received from the layout and style indexes. What I'm wondering is if
> this lookup which would be a Lucene search will be fast enough?
> >>>>>
> >>>>> Would it be best to make Lucene queries for the relevant layout
> style values required by the renderers ahead of rendering time and have the
> query results placed into the most performant collection (map/array) so
> renderer lookup would be as fast as possible? Or can Lucene handle many
> individual lookup queries fast enough so rendering is quick?
> >>>>>
> >>>>> Best regards from Canada,
> >>>>>
> >>>>> Thom
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
> > --------------------------
> > Grant Ingersoll
> >
> >
> > Search the Lucene ecosystem using Solr/Lucene:
> >
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message