lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: DIH full-import memory issue
Date Tue, 11 May 2010 21:47:51 GMT
Also, which JDBC driver is this? There are quirks with various
drivers, which should be documented on the DataImportHandler page.

On Mon, May 10, 2010 at 9:47 PM, caman <aboxfortheotherstuff@gmail.com> wrote:
>
> This may help:
>
> batchSize : The batchsize used in jdbc connection
>
>
>
> http://wiki.apache.org/solr/DataImportHandler#Configuring_DataSources
>
>
>
>
>
>
>
>
>
> From: Geek Gamer [via Lucene]
> [mailto:ml-node+809069-2054572211-124354@n3.nabble.com]
> Sent: Monday, May 10, 2010 9:42 PM
> To: caman
> Subject: DIH full-import memory issue
>
>
>
> Hi,
>
> I am facing issues with DIH fullimport,
>
> I have a database with 3 million records that will translate into index size
>
> of 6GB.
>
> When I am trying to do full import I am getting out of memory error like :
>
> INFO: Starting Full Import
> May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.SolrWriter
> readIndexerProperties
> WARNING: Unable to read: dataimport.properties
> May 10, 2010 11:44:06 PM org.apache.solr.update.DirectUpdateHandler2
> deleteAll
> INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
> May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=/home/search/SOLR/solr/data/index,segFN=segments_1,version=127354
> 9043650,generation=1,filenames=[segments_1]
> May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1273549043650
> May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
>
> call
> INFO: Creating a connection for entity offer with URL:
> jdbc:mysql://domU-12-31-39-10-59-01.compute-1.internal/jounce1
> May 10, 2010 11:44:07 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
>
> call
> INFO: Time taken for getConnection(): 301
>
>
>
> Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.newValueIterator(HashMap.java:843)
> at java.util.HashMap$Values.iterator(HashMap.java:910)
> at
> org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.jav
> a:180)
> at
> org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.j
> ava:36)
> at
> org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:1
> 44)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> May 10, 2010 11:54:54 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> SEVERE: Full Import failed
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
> 424)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242
> )
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
> va:331)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389
> )
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
> at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
> at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:734)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(J
> dbcDataSource.java:246)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
> a:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
> a:39)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityPro
> cessor.java:58)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProce
> ssor.java:71)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProc
> essorWrapper.java:237)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
> 357)
> ... 5 more
> May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: start rollback
> May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2
> rollback
> INFO: end_rollback
>
>
>
>
> I tried allocating 4 Gigs of memory to the VM but no luck.
> Are the records cached before indexing or streamed?
> any pointers to documents?
>
> thanks in anticipation,
> umar
>
>
>
>  _____
>
> View message @
> http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809
> 069.html
> To start a new topic under Solr - User, email
> ml-node+472068-464289649-124354@n3.nabble.com
> To unsubscribe from Solr - User, click
> < (link removed)
> GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx>  here.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809076.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message