lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Importing CSV file slow/crashes
Date Tue, 06 Oct 2009 22:22:56 GMT
On Tue, Oct 6, 2009 at 1:06 PM, Nasseam Elkarra <nasseam@bodukai.com> wrote:
> I had a dev build of 1.4 from 5/1/2009 and importing a 20K row took less
> than a minute. Updating to the latest as of yesterday, the import is really
> slow and I had to cancel it after a half hour. This prevented me from
> upgrading a few months ago as well.

I haven't had any success at replicating this problem.

I just tried a 100K row CSV file, consisting of an id and a few text
fields.  The total size of the file is 79MB.

On trunk (today): 22 seconds to index, another 5-7 secons to commit
5/21 version: 28 seconds to index, another 8 seconds to commit

Then I modified the 5/1 schema to closer match the trunk schema
(removing defaults, copyfields that could slow things down).
Modified 5/1 version: 25 seconds to index, another 8 seconds to commit

I only did 2 runs with trunk and 2 with one from 5/1, so the accuracy
is probably low... but good enough to see there wasn't a problem in
this test.

We really need more info to help reproduce this.
Are you using stock solr?  Do you have any custom plugins, analyzers,
token filters, etc?

You're going to need to provide something so others can reproduce this.

-Yonik
http://www.lucidimagination.com

Mime
View raw message