lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Constantijn Visinescu <baeli...@gmail.com>
Subject Re: DIH out of memory exception
Date Wed, 28 Oct 2009 08:18:31 GMT
Does this help?

http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F

On Wed, Oct 28, 2009 at 12:38 AM, William Pierce <evalsinca@hotmail.com>wrote:

> Hi, Gilbert:
>
> Thanks for your tip!  I just tried it.  Unfortunately, it does not work for
> me.  I still get the OOM exception.
>
> How large was your dataset?  And what were your machine specs?
>
> Cheers,
>
> - Bill
>
> --------------------------------------------------
> From: "Gilbert Boyreau" <gboyreau@andevsol.com>
> Sent: Tuesday, October 27, 2009 11:54 AM
> To: <solr-user@lucene.apache.org>
> Subject: Re: DIH out of memory exception
>
>
>  Hi,
>>
>> I got the same problem using DIH with a large dataset in MySql database.
>>
>> Following :
>> http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-implementation-notes.html
>> ,
>> and looking at the java code, it appears that DIH use PreparedStatement in
>> the JdbcDataSource.
>>
>> I set the batchsize parameter to -1 and it solved my problem.
>>
>> Regards.
>> Gilbert.
>>
>> William Pierce a écrit :
>>
>>> Folks:
>>>
>>> My db contains approx 6M records -- on average each is approx 1K bytes.
>>> When I use the DIH,  I reliably get an OOM exception.   The machine has 4 GB
>>> ram,  my tomcat is set to use max heap of 2G.
>>> The option of increasing memory is not tenable coz as the number of
>>> documents grows I will be back in this situation.
>>> Is there a way to batch the documents?  I tried setting the batchsize
>>> parameter to 500 on the <dataSource> tag where I specify the jdbc
>>> parameters.   This had no effect.
>>>
>>> Best,
>>>
>>> - Bill
>>>
>>>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message