sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shuaishuai Nie" <shuai...@microsoft.com>
Subject Re: Review Request 11320: Review request for SQOOP-1056 "Implement connection resiliency in Sqoop using pluggable failure handlers"
Date Fri, 31 Jan 2014 01:24:34 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11320/
-----------------------------------------------------------

(Updated Jan. 31, 2014, 1:24 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Changes
-------

Combine patch from SQOOP-1056 and SQOOP-1057


Bugs: SQOOP-1056
    https://issues.apache.org/jira/browse/SQOOP-1056


Repository: sqoop-trunk


Description
-------

Implement a pluggable way for handling connection failures, and/or intermittent errors in
Sqoop. This is especially crucial in environments where the probability of connections getting
reset or throttled is high.
In case of intermittent failures in Sqoop, due to connection losses or server throttling,
Sqoop does not recover from those failures. As a result, the running Sqoop task would eventually
fail, and a new task is started. In those cases, Sqoop does not always guarantee that tasks
can safely be restarted. For example, if part of the records is already committed to the database,
then restarting the task would result in some failures like primary key violations. Even for
Sqoop jobs which commit the records only at the end of the task, any failures towards the
end of the task would involve reprocessing the whole range of split owned by the task, and
any progress is lost. (More detail please refer to the document attached at SQOOP-1056)


Diffs (updated)
-----

  build.xml 74150e6 
  ivy.xml c00d30c 
  ivy/libraries.properties 6b9b2d7 
  src/java/org/apache/sqoop/manager/SQLServerManager.java eea5a1d 
  src/java/org/apache/sqoop/manager/SqlManager.java 8248263 
  src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 7b9e4d2 
  src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 4b44244 
  src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java 2c08f12 
  src/test/aop/build/aop.xml PRE-CREATION 
  src/test/aop/org/apache/sqoop/fi/FiConfig.java PRE-CREATION 
  src/test/aop/org/apache/sqoop/fi/ProbabilityModel.java PRE-CREATION 
  src/test/aop/org/apache/sqoop/mapreduce/SqlServerExportAspect.aj PRE-CREATION 
  src/test/aop/org/apache/sqoop/mapreduce/db/SqlServerImportAspect.aj PRE-CREATION 
  src/test/fi-site.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/11320/diff/


Testing
-------

Tested with fault injection on SQL server integration tests from SQOOP-1035


Thanks,

Shuaishuai Nie


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message