lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: Solr filter using data from the database
Date Tue, 20 Nov 2012 09:25:26 GMT
Hello Colleagues,
Recently I had talk at ApacheCon about this problem. Both proposed
approaches are definitely work. Frequent updates
http://goo.gl/xGPMUsometimes cost too much. Filters
http://goo.gl/mMvRQ works slow starting from thousand of keys and might
have low hit ratio. One of the promising approach is FunctionRangeQuery +
ExternalFileField, the current implementation requires commit for reload
that's a drawback. Here is the possible solution for this
https://issues.apache.org/jira/browse/SOLR-4085 .

Good Luck


On Sat, Nov 17, 2012 at 6:32 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> I'm actually not sure what Wunder is suggesting, but here is another way.
> Have an external app that talks to the DB either on demand or every N
> minutes/hours.  When it talks to the DB it gets all merchants whose
> visibility flag was changed one way or the other since the last time the
> app checked.  Then you can simply delete all products for those merchants
> whose products are supposed to be hidden, and reindex all those whose flag
> was switched to visible.  Depending on the numbers this may or may not work
> well, but it's super simple and Solr and the DB don't know about each
> other.
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm/index.html
> Search Analytics - http://sematext.com/search-analytics/index.html
>
>
>
>
> On Fri, Nov 16, 2012 at 7:32 PM, Hayden Muhl <haydenmuhl@gmail.com> wrote:
>
> > I am working on migrating our system from Lucene to Solr, and my boss
> and I
> > are at an impasse over an architectural issue. Here's the basic setup.
> >
> > We index products from multiple retailers, and allow people to search
> > across all retailers for specific products. It is a regular occurrence
> for
> > us to disable or deactivate a retailer (on the order of once a day). What
> > that means is that when someone does a search, we will not show results
> for
> > products sold by deactivated retailers. The list of active/inactive
> > retailers is maintained in a table in our SQL database. We have to
> maintain
> > this functionality when we move to Solr, but we can't agree on how to
> > implement this in Solr.
> >
> > Currently, we load a new Lucene product index once every two hours. Every
> > time we load a new index, we run a SQL query to find the current list of
> > active retailers, and construct a filter based on that list.
> >
> > My boss wants to essentially do the same thing we do now. Implement a
> > custom filter that makes a call to the database to retrieve the list of
> > retailers, caches that list for some period of time, then refreshes
> itself
> > from time to time with another call to the database. I find it strange
> > having a dependency between Solr and the database like that, because it
> > would require a running database being present in order to even start
> Solr.
> > I am new to Solr, so I don't have any alternative solutions.
> >
> > tl;dr, We need to construct a Solr filter based on data stored in a
> > database. What's the best way to get that data from the database into
> Solr
> > and keep it updated?
> >
> > - Hayden
> >
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
 <mkhludnev@griddynamics.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message