sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abraham Elmahrek" <...@cloudera.com>
Subject Re: Review Request 23531: Sqoop2: From/To: Refactor connector interface
Date Tue, 22 Jul 2014 21:46:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23531/
-----------------------------------------------------------

(Updated July 22, 2014, 9:46 p.m.)


Review request for Sqoop, Gwen Shapira, Hari Shreedharan, and Jarek Cecho.


Changes
-------

Added list of most important modification.


Bugs: SQOOP-1376
    https://issues.apache.org/jira/browse/SQOOP-1376


Repository: sqoop-sqoop2


Description
-------

commit 8327ff3b6432f2d9358991b694945411fff26025
Author: Abraham Elmahrek <abraham@elmahrek.com>
Date:   Wed Jul 9 12:42:17 2014 -0700

    Sqoop2: From/To: Refactor connector interface
    
    Various components have been hacked together to compile.
    Their working order are to be addressed in follow up Jiras.
    This includes, but is not limited to:
    Repository, Execution Engine, Submission Engine, Core,
    Server, and the shell.
    
    ConectorBean and JobBean have a simple design. Improvements
    are to be addressed in follow up Jiras.
    
    MConnector now has 2 MJobForms instances.
    There is a MJobForms instance for the "from" connector and
    a MJobForms instance for the "to" connector.
    MConnector still has only one MConnectionForms.
    The information needed when extracting using one connector may
    be different than the information needed when loading using
    the same connector. Example: GenericJdbcConnector can use
    a staging table when it is the connector being used to load
    data TO. It does not need a staging table when extracting data
    FROM.
    Since Connections define how Sqoop connects to a database,
    there shouldn't be any variation based on FROM/TO.
    
    MJob objects must now be aware of Connectors that it is
    extracting data FROM and sending data TO. It also keeps
    track of Connection objects for both Connectors.
    
    SPI changed to allow FROM/TO. Initializers, laoders, etc.
    were not changed, just the classes that extend
    CallbackBase.

:100644 100644 05ea6d6... d1f7a1d... M  client/src/main/java/org/apache/sqoop/client/SqoopClient.java
:100644 100644 cbe049a... f6b2de4... M  common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
:100644 100644 eb79f98... e234908... M  common/src/main/java/org/apache/sqoop/json/FrameworkBean.java
:100644 100644 1555bd5... 58c53d3... M  common/src/main/java/org/apache/sqoop/json/JobBean.java
:000000 100644 0000000... c9a1c42... A  common/src/main/java/org/apache/sqoop/json/JobValidationBean.java
:100644 100644 43fad27... 117ecdc... M  common/src/main/java/org/apache/sqoop/model/MConnector.java
:100644 100644 c742459... 580db9c... M  common/src/main/java/org/apache/sqoop/model/MFramework.java
:100644 100644 849168d... 155f246... M  common/src/main/java/org/apache/sqoop/model/MJob.java
:100644 100644 f697023... 8505483... M  common/src/main/java/org/apache/sqoop/model/MJobForms.java
:100644 100644 e0da80f... ab33568... M  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
:100644 100644 0c5f6e1... 3e2f89a... M  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcValidator.java
:100644 100644 17215f0... 346b625... M  connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java
:100644 100644 b80de7f... 9cb6244... M  core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java
:100644 100644 505121c... fe21a3f... M  core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java
:100644 100644 f5f6a36... 0bd4947... M  core/src/main/java/org/apache/sqoop/framework/FrameworkValidator.java
:100644 100644 e052584... b5bd3a2... M  core/src/main/java/org/apache/sqoop/framework/JobManager.java
:100644 100644 53d0039... 9e8c07c... M  core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
:000000 100644 0000000... 7c653bf... A  core/src/main/java/org/apache/sqoop/framework/configuration/JobConfiguration.java
:100644 100644 ecf5004... 95a8bd5... M  core/src/main/java/org/apache/sqoop/repository/Repository.java
:100644 100644 5c0a027... 6352bb5... M  execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
:100644 100644 bd11323... 96ecf32... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java
:100644 100644 e1a95a7... 0edcb7a... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
:100644 100644 6891258... 516c80a... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java
:100644 100644 92de37e... 37f416d... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
:100644 100644 7dedee9... b5adace... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
:100644 100644 5bce3a9... f002db6... M  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
:100644 100644 fcbb475... 1a77360... M  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java
:100644 100644 7042a53... fa16360... M  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
:100644 100644 362ba79... a9394b6... M  server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java
:100644 100644 9cce17b... 99edb6b... M  shell/src/main/java/org/apache/sqoop/shell/CloneConnectionFunction.java
:100644 100644 f80552c... 11878f1... M  shell/src/main/java/org/apache/sqoop/shell/CloneJobFunction.java
:100644 100644 598adbc... e063f9a... M  shell/src/main/java/org/apache/sqoop/shell/CreateJobFunction.java
:100644 100644 54d8e9a... 51e189f... M  shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
:100644 100644 6f07c32... ee1c8c4... M  shell/src/main/java/org/apache/sqoop/shell/DisableConnectionFunction.java
:100644 100644 9256712... 71295fa... M  shell/src/main/java/org/apache/sqoop/shell/EnableConnectionFunction.java
:100644 100644 6e5c9b5... dfaa90e... M  shell/src/main/java/org/apache/sqoop/shell/ShowConnectionFunction.java
:100644 100644 9a5386c... b69b726... M  shell/src/main/java/org/apache/sqoop/shell/ShowJobFunction.java
:100644 100644 8fa1bda... 791cc03... M  shell/src/main/java/org/apache/sqoop/shell/UpdateConnectionFunction.java
:100644 100644 b060bb4... 43baf28... M  shell/src/main/java/org/apache/sqoop/shell/UpdateJobFunction.java
:100644 100644 475f41c... 5765260... M  shell/src/main/java/org/apache/sqoop/shell/core/Constants.java
:100644 100644 56e0b4e... c2f9e22... M  shell/src/main/java/org/apache/sqoop/shell/utils/FormDisplayer.java
:100644 100644 c491ae5... e4b63ee... M  shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
:100644 100644 aa118e1... 7bc6154... M  shell/src/main/java/org/apache/sqoop/shell/utils/JobDynamicFormOptions.java
:100644 100644 2becc56... 7d92ef6... M  spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java
:100644 000000 cdaa623... 0000000... D  spi/src/main/java/org/apache/sqoop/job/etl/Exporter.java
:000000 100644 0000000... 9b8d76f... A  spi/src/main/java/org/apache/sqoop/job/etl/From.java
:100644 000000 d4c9e70... 0000000... D  spi/src/main/java/org/apache/sqoop/job/etl/Importer.java
:000000 100644 0000000... a791945... A  spi/src/main/java/org/apache/sqoop/job/etl/To.java
:100644 100644 cf0b4aa... 9b791f8... M  spi/src/main/java/org/apache/sqoop/validation/Validator.java
:100644 100644 6fc485b... 54071f2... M  submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java


Diffs
-----

  client/src/main/java/org/apache/sqoop/client/SqoopClient.java 05ea6d6 
  common/src/main/java/org/apache/sqoop/common/ConnectorType.java PRE-CREATION 
  common/src/main/java/org/apache/sqoop/json/ConnectorBean.java cbe049a 
  common/src/main/java/org/apache/sqoop/json/FrameworkBean.java eb79f98 
  common/src/main/java/org/apache/sqoop/json/JobBean.java 1555bd5 
  common/src/main/java/org/apache/sqoop/json/JobValidationBean.java PRE-CREATION 
  common/src/main/java/org/apache/sqoop/model/MConnector.java 43fad27 
  common/src/main/java/org/apache/sqoop/model/MFramework.java c742459 
  common/src/main/java/org/apache/sqoop/model/MJob.java 849168d 
  common/src/main/java/org/apache/sqoop/model/MJobForms.java f697023 
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
e0da80f 
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcValidator.java
0c5f6e1 
  connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java
17215f0 
  core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java b80de7f 
  core/src/main/java/org/apache/sqoop/framework/ExecutionEngine.java f43942d 
  core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 505121c 
  core/src/main/java/org/apache/sqoop/framework/FrameworkValidator.java f5f6a36 
  core/src/main/java/org/apache/sqoop/framework/JobManager.java e052584 
  core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java 53d0039 
  core/src/main/java/org/apache/sqoop/framework/configuration/JobConfiguration.java PRE-CREATION

  core/src/main/java/org/apache/sqoop/repository/Repository.java ecf5004 
  execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
5c0a027 
  execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java 7fd9a01 
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java bd11323

  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java e1a95a7

  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 6891258

  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 92de37e 
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
7dedee9 
  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
5bce3a9 
  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java
fcbb475 
  repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
7042a53 
  server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java 362ba79 
  shell/src/main/java/org/apache/sqoop/shell/CloneConnectionFunction.java 9cce17b 
  shell/src/main/java/org/apache/sqoop/shell/CloneJobFunction.java f80552c 
  shell/src/main/java/org/apache/sqoop/shell/CreateJobFunction.java 598adbc 
  shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java 54d8e9a 
  shell/src/main/java/org/apache/sqoop/shell/DisableConnectionFunction.java 6f07c32 
  shell/src/main/java/org/apache/sqoop/shell/EnableConnectionFunction.java 9256712 
  shell/src/main/java/org/apache/sqoop/shell/ShowConnectionFunction.java 6e5c9b5 
  shell/src/main/java/org/apache/sqoop/shell/ShowJobFunction.java 9a5386c 
  shell/src/main/java/org/apache/sqoop/shell/UpdateConnectionFunction.java 8fa1bda 
  shell/src/main/java/org/apache/sqoop/shell/UpdateJobFunction.java b060bb4 
  shell/src/main/java/org/apache/sqoop/shell/core/Constants.java 475f41c 
  shell/src/main/java/org/apache/sqoop/shell/utils/FormDisplayer.java 56e0b4e 
  shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java c491ae5 
  shell/src/main/java/org/apache/sqoop/shell/utils/JobDynamicFormOptions.java aa118e1 
  spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 2becc56 
  spi/src/main/java/org/apache/sqoop/job/etl/Exporter.java cdaa623 
  spi/src/main/java/org/apache/sqoop/job/etl/From.java PRE-CREATION 
  spi/src/main/java/org/apache/sqoop/job/etl/Importer.java d4c9e70 
  spi/src/main/java/org/apache/sqoop/job/etl/To.java PRE-CREATION 
  spi/src/main/java/org/apache/sqoop/validation/Validator.java cf0b4aa 
  submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
6fc485b 

Diff: https://reviews.apache.org/r/23531/diff/


Testing (updated)
-------

N/A

This patch should break just about every thing. Most important modifications:
- MConnector
- MJob
- MConnection
- MFramework
- From (Exporter)
- To (Importer)
- JobManager


Thanks,

Abraham Elmahrek


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message