lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From caman <aboxfortheotherst...@gmail.com>
Subject RE: DIH full-import memory issue
Date Tue, 11 May 2010 04:47:23 GMT

This may help:

batchSize : The batchsize used in jdbc connection

 

http://wiki.apache.org/solr/DataImportHandler#Configuring_DataSources

 

 

 

 

From: Geek Gamer [via Lucene]
[mailto:ml-node+809069-2054572211-124354@n3.nabble.com] 
Sent: Monday, May 10, 2010 9:42 PM
To: caman
Subject: DIH full-import memory issue

 

Hi, 

I am facing issues with DIH fullimport, 

I have a database with 3 million records that will translate into index size

of 6GB. 

When I am trying to do full import I am getting out of memory error like : 

INFO: Starting Full Import 
May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.SolrWriter 
readIndexerProperties 
WARNING: Unable to read: dataimport.properties 
May 10, 2010 11:44:06 PM org.apache.solr.update.DirectUpdateHandler2 
deleteAll 
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX 
May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy onInit 
INFO: SolrDeletionPolicy.onInit: commits:num=1 
commit{dir=/home/search/SOLR/solr/data/index,segFN=segments_1,version=127354
9043650,generation=1,filenames=[segments_1] 
May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy 
updateCommits 
INFO: newest commit = 1273549043650 
May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.JdbcDataSource$1

call 
INFO: Creating a connection for entity offer with URL: 
jdbc:mysql://domU-12-31-39-10-59-01.compute-1.internal/jounce1 
May 10, 2010 11:44:07 PM org.apache.solr.handler.dataimport.JdbcDataSource$1

call 
INFO: Time taken for getConnection(): 301 



Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space 
at java.util.HashMap.newValueIterator(HashMap.java:843) 
at java.util.HashMap$Values.iterator(HashMap.java:910) 
at 
org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.jav
a:180) 
at 
org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.j
ava:36) 
at 
org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:1
44) 
at java.util.TimerThread.mainLoop(Timer.java:512) 
at java.util.TimerThread.run(Timer.java:462) 
May 10, 2010 11:54:54 PM org.apache.solr.handler.dataimport.DataImporter 
doFullImport 
SEVERE: Full Import failed 
org.apache.solr.handler.dataimport.DataImportHandlerException: 
java.lang.OutOfMemoryError: Java heap space 
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
424) 
at 
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242
) 
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) 
at 
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja
va:331) 
at 
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389
) 
at 
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)

Caused by: java.lang.OutOfMemoryError: Java heap space 
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621) 
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398) 
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816) 
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467) 
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510) 
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746) 
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135) 
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536) 
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465) 
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:734) 
at 
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(J
dbcDataSource.java:246) 
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
a:210) 
at 
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav
a:39) 
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityPro
cessor.java:58) 
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProce
ssor.java:71) 
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProc
essorWrapper.java:237) 
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:
357) 
... 5 more 
May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2 
rollback 
INFO: start rollback 
May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2 
rollback 
INFO: end_rollback 




I tried allocating 4 Gigs of memory to the VM but no luck. 
Are the records cached before indexing or streamed? 
any pointers to documents? 

thanks in anticipation, 
umar 



  _____  

View message @
http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809
069.html 
To start a new topic under Solr - User, email
ml-node+472068-464289649-124354@n3.nabble.com 
To unsubscribe from Solr - User, click
< (link removed) 
GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx>  here. 

 


-- 
View this message in context: http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809076.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message