lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Scaling DataImportHandler
Date Fri, 22 Jan 2016 20:48:49 GMT
On 1/22/2016 10:29 AM, Brian Narsi wrote:
> What are the various ways DataImportHandler can be scaled?

I'm not very familiar with how DIH interacts with SolrCloud.  I know you 
can use it with SolrCloud, but nothing else.  Assuming you're not 
running SolrCloud, the following information will apply:

The only way I know of to scale DIH performance on a single Solr core is 
to define multiple DIH handlers, each of which imports a subset of your 
data.  Then you can run multiple imports simultaneously.  A single DIH 
handler can only run one import at a time, and it will be single-threaded.

If you've got a sharded index, each shard would be in its own core, and 
therefore each one would have its own instance of your dataimport 
handler with its own config, so you can run an import on each shard 
simultaneously.  This is what happens on my index when I need to fully 
rebuild it.

Thanks,
Shawn


Mime
View raw message