lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qwerky <neil.j.tay...@hmv.co.uk>
Subject Delta import processing duration
Date Thu, 22 Jul 2010 16:33:30 GMT

I'm using Solr to index data from our data warehouse. The data is imported
through text files. I've written a custom FileImportDataImportHandler that
extends DataSource and it works fine - I've tested it with 280,000 records
and it manages to build the index in about 3 minutes. My problem is that
doing a delta update seems to take a really long time.

I've written a custome FileUpdateDataImportHandler which takes two files,
one for deletes and one fore updates. I've tested with an update file
containing 18,000 records and a delete file containing 30 records - my
custom handler whizzed through them in a few seconds but the page at
/solr/admin/dataimport.jsp says the command is still running (its been
running nearly an hour).

What's taking so long? Could there be some kind of inefficiency in the way
my update handler works?
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Delta-import-processing-duration-tp987562p987562.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message