lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Barry Harding <ba...@systemax.co.uk>
Subject dataimport.properties
Date Mon, 19 Sep 2011 10:36:11 GMT
Hi I am currently using the DIH to connect to and import data from a MS SQL Server, and in
general doing full, delta or deletes seems to work perfectly.

The issue is that I spotted some errors being logged in the tomcat logs for SOLR which are
:

19-Sep-2011 07:45:25 org.apache.solr.common.SolrException log
SEVERE: Exception in entity : product:org.apache.solr.handler.dataimport.DataImportHandlerException:
com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 125) was deadlocked
on lock resources with another process and has been chosen as the deadlock victim. Rerun the
transaction.
	at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:64)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:339)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:228)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:262)
	at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:77)
	at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
	at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:238)
	at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:591)
	at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:302)
	at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:178)
	at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
	at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 125) was
deadlocked on lock resources with another process and has been chosen as the deadlock victim.
Rerun the transaction.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:213)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:4713)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1671)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:944)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:331)
	... 11 more

19-Sep-2011 07:45:25 org.apache.solr.handler.dataimport.DocBuilder doDelta
INFO: Delta Import completed successfully
19-Sep-2011 07:45:25 org.apache.solr.handler.dataimport.DocBuilder finish


Now this SQL error I can deal with and I will probably switch to snapshot Isolation as these
are constantly updated tables, but my issue is not the sql error but the fact that the delta
import still reported that it had imported successfully and still wrote out the last updated
time to the dataimport.properties file, so the next time it ran it missed a bunch of documents
that should have been indexed.

If it had failed and just rolled back the changes and not updated the dataimport.properties
file it would (assuming no more deadlocks) caught all of the missed documents on the next
delta import.

My connection to MS SQL is using the "responseBuffering=adaptive" setting to reduce memory
overhead, So I guess what I am asking is there any way I can cause the DIH to roll back the
import if an error occurs and to not update the "dataimport.properties" file.

Any help or suggestions would be appreciated

Thanks

Barry H
 
DISCLAIMER: This email and its attachments may be confidential and are intended solely for
the use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Misco UK Ltd. Any unauthorised
use or dissemination of this communication is strictly prohibited. If you have received this
communication in error, please immediately notify the sender by return e-mail message and
delete all copies of the original communication. Thank you for your cooperation. Misco UK
Ltd, registered in Scotland Number 114143. Registered Office: Caledonian Exchange, 19a Canning
Street, Edinburgh EH3 8EG. Telephone +44 (0)1933 686000. This e-mail message has been scanned
by CA Gateway Security.

Mime
View raw message