lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Klostermeyer, Michael" <>
Subject RE: DIH - unable to ADD individual new documents
Date Tue, 03 Jul 2012 22:07:55 GMT
I haven't, but will consider those alternatives.  I think right now I'm going to go w/ a hybrid
approach, meaning my scheduled and full updates will continue to use the DIH, as those seem
to work really well.  My NTR indexing needs will be handled via the JSON processor.  For individual
updates this will enable me to utilize an existing ORM infrastructure fairly easily (famous
last words, I know).

Thanks for the help, as always.


-----Original Message-----
From: Erick Erickson [] 
Sent: Tuesday, July 03, 2012 2:58 PM
Subject: Re: DIH - unable to ADD individual new documents


Have you considered using one (or several) SolrJ clients to do your indexing? That can give
you a finer control granularity than DIH. Or even do your NRT with SolrJ or....

Here's an example program, you can take out the Tika stuff pretty easily..


On Tue, Jul 3, 2012 at 3:35 PM, Klostermeyer, Michael <>
> Well that little bit of knowledge changes things for me, doesn't it?  I appreciate your
response very much.  Without knowing that about the DIH, I attempted to have my DIH handler
handle all circumstances, namely the "batch", scheduled job, and immediate/NRT indexing. 
Looks like I'm going to have to severely re-think that strategy.
> Thanks again...and if anyone has any further input how I can best/most efficiently accomplish
all 3 above, please let me know.
> Mike
> -----Original Message-----
> From: Dyer, James []
> Sent: Tuesday, July 03, 2012 1:12 PM
> To:
> Subject: RE: DIH - unable to ADD individual new documents
> A DIH request handler can only process one "run" at a time.  So if DIH is still in process
and you kick off a new DIH "full-import" it will silently ignore the new command.  To have
more than one DIH "run" going at a time it is necessary to configure more than one handler
instance in sorlconfig.xml.  But even then you'll have to be careful to find one that is free
before trying to use it.
> Regardless, to do what you want, you'll need to poll the DIH response screen to be sure
it isn't running before starting a new one.  It would be simplest to leave it with just 1
DIH handler in solrconfig.xml.  If you've got to have an undefined # of concurrent updates
going at once you're best off to not use DIH.
> Perhaps a better usage pattern for which DIH was designed for is to put the doc id's
in an update table with a timestamp.  Have your queries join to the update table "where timestamp
> ${dih.last_index_time}".  Set up crontab or whatever to kick off DIH every so often.
 If the prior run is still in progress, it will just skip that run, but because we're dealing
with timestamps that get written automatically when DIH finishes, you will only experience
a delayed update, not a lost update.  By batching your updates like this you will also have
fewer commits, which will be beneficial for performance all around.
> Of course if you're trying to do this with the near-real-time functionality batching
isn't your answer.  But DIH isn't designed at all to work well with NRT either...
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
> -----Original Message-----
> From: Klostermeyer, Michael []
> Sent: Tuesday, July 03, 2012 1:55 PM
> To:
> Subject: RE: DIH - unable to ADD individual new documents
> Some interesting findings over the last hours, that may change the context of this discussion...
> Due to the nature of the application, I need the ability to fire off individual "ADDs"
on several different entities at basically the same time.  So, I am making 2-4 Solr ADD calls
within 100ms of each other.  While troubleshooting this, I found that if I only made 1 Solr
ADD call (ignoring the other entities), it updated the index as expected.  However, when all
were fired off, proper indexing did not occur (at least on one of the entities) and no errors
were logged.  I am still attempting to figure out if ALL of the 2-4 entities failed to ADD,
or if some failed and others succeeded.
> So...does this have something to do with Solr's index/message queuing (v3.5)?  How does
Solr handle these types of rapid requests, and even more important, how do I get the status
of an individual DIH call vs simply the status of the "latest" call at /dataimport?
> Mike
> -----Original Message-----
> From: Gora Mohanty []
> Sent: Monday, July 02, 2012 10:02 PM
> To:
> Subject: Re: DIH - unable to ADD individual new documents
> On 3 July 2012 07:54, Klostermeyer, Michael <> wrote:
>> I should add that I am using the full-import command in all cases, and setting clean=false
for the individual adds.
> What does the data-import page report at the end of the full-import, i.e., how many documents
were indexed?
> Are there any error messages in the Solr logs? Please share with us your DIH configuration
file, and Solr schema.xml.
> Regards,
> Gora

View raw message