lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Pierce" <evalsi...@hotmail.com>
Subject Re: DIH out of memory exception
Date Tue, 27 Oct 2009 23:38:30 GMT
Hi, Gilbert:

Thanks for your tip!  I just tried it.  Unfortunately, it does not work for 
me.  I still get the OOM exception.

How large was your dataset?  And what were your machine specs?

Cheers,

- Bill

--------------------------------------------------
From: "Gilbert Boyreau" <gboyreau@andevsol.com>
Sent: Tuesday, October 27, 2009 11:54 AM
To: <solr-user@lucene.apache.org>
Subject: Re: DIH out of memory exception

> Hi,
>
> I got the same problem using DIH with a large dataset in MySql database.
>
> Following : 
> http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-implementation-notes.html,
> and looking at the java code, it appears that DIH use PreparedStatement in 
> the JdbcDataSource.
>
> I set the batchsize parameter to -1 and it solved my problem.
>
> Regards.
> Gilbert.
>
> William Pierce a écrit :
>> Folks:
>>
>> My db contains approx 6M records -- on average each is approx 1K bytes. 
>> When I use the DIH,  I reliably get an OOM exception.   The machine has 4 
>> GB ram,  my tomcat is set to use max heap of 2G.
>> The option of increasing memory is not tenable coz as the number of 
>> documents grows I will be back in this situation.
>> Is there a way to batch the documents?  I tried setting the batchsize 
>> parameter to 500 on the <dataSource> tag where I specify the jdbc 
>> parameters.   This had no effect.
>>
>> Best,
>>
>> - Bill
>>
>
> 

Mime
View raw message