lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Stuart <>
Subject Re: Importing large datasets
Date Wed, 02 Jun 2010 19:00:02 GMT
How long does it take to do a grab of all the data via SQL? I found by  
denormalizing the data into a lookup table meant that I was able to  
index about 300k rows of similar data size with dih regex spilting on  
some fields in about 8mins I know it's not quite the scale bit with  

David Stuar

On 2 Jun 2010, at 17:58, Blargy <> wrote:

>> One thing that might help indexing speed - create a *single* SQL  
>> query
>> to grab all the data you need without using DIH's sub-entities, at
>> least the non-cached ones.
> Not sure how much that would help. As I mentioned that without the  
> item
> description import the full process takes 4 hours which is bearable.  
> However
> once I started to import the item description which is located on a  
> separate
> machine/database the import process exploded to over 24 hours.
> -- 
> View this message in context:
> Sent from the Solr - User mailing list archive at

View raw message