lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From marotosg <marot...@gmail.com>
Subject DataImportHandler OutOfMemory Mysql
Date Sat, 01 Apr 2017 22:17:42 GMT
Hi,

I am trying to load a big table into Solr using DataImportHandler and Mysql. 
I am getting OutOfMemory error because Solr is trying to load the full
table. I have been reading different posts and tried batchSize="-1". 
https://wiki.apache.org/solr/DataImportHandlerFaq

Do you have any idea what could be the issue?
Completely lost here.

Solr.6.4.1
mysql-connector-java-5.1.41-bin.jar

data-config 

<dataSource type="JdbcDataSource" 
            driver="com.mysql.jdbc.Driver"
            url="jdbc:mysql://188.68.190.85:3306/jobsdb" 
            user="suer" 
            password="passowrd"/>
<document>
  <entity name="jobsearch"  
    pk="id"
	batchSize="-1"
    query="select * from job"
    deltaImportQuery="SELECT * from job WHERE id='${dih.delta.id}'"
    deltaQuery="SELECT id FROM job  WHERE updated_at >
'${dih.last_index_time}'"
    >
     <field column="job_id" name="JobID"/>
     <field column="position" name="Position"/>    
     <field column="employment_type" name="EmploymentType"/>    
	 <field column="description" name="Description"/>
     <field column="category" name="Category"/>  
	 <field column="apply_url" name="ApplyUrl"/>
     <field column="description_url" name="DescriptionUrl"/>  	
	 <field column="company" name="Company"/>  	
	 <field column="city" name="City"/>  	
	 <field column="country_subdivision1" name="CountrySubdivision1"/>  	
	 <field column="country_subdivision2" name="CountrySubdivision2"/>  	
	 <field column="country" name="Country"/>  	
     <field column="source" name="Source"/>
	 <field column="created_at" name="CreatedAt"/>
	 <field column="updated_at" name="UpdatedAt"/>	  
  </entity>
</document>
</dataConfig>

Thanks
Sergio



--
View this message in context: http://lucene.472066.n3.nabble.com/DataImportHandler-OutOfMemory-Mysql-tp4327982.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message