sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abraham Elmahrek" <...@cloudera.com>
Subject Re: Review Request 16812: SQOOP-777. Sqoop2: Pluggable Intermediate Data Format
Date Sun, 27 Jul 2014 05:51:12 GMT


> On July 19, 2014, 10:03 p.m., Jarek Cecho wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java, line
27
> > <https://reviews.apache.org/r/16812/diff/5/?file=604300#file604300line27>
> >
> >     Didn't we wanted to use the Configurable interface to skip the string encoding
here?
> 
> Abraham Elmahrek wrote:
>     Yessir, but we've decided to do this in a follow up Jira when Hadoop 2.5 is released.
> 
> Jarek Cecho wrote:
>     Make sense, could you then create the follow up JIRA?

https://issues.apache.org/jira/browse/SQOOP-1349


- Abraham


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16812/#review48182
-----------------------------------------------------------


On July 22, 2014, 7:50 a.m., Abraham Elmahrek wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16812/
> -----------------------------------------------------------
> 
> (Updated July 22, 2014, 7:50 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-777
>     https://issues.apache.org/jira/browse/SQOOP-777
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Previous review from hari is in first iteration of this review only.
> 
> commit 1c5122611fbd4a46a629421f7c55746cfc14f136
> Author: Hari Shreedharan <hshreedharan@apache.org>
> Date:   Fri Jan 10 15:07:08 2014 -0800
> 
>     SQOOP-777. Sqoop2: Pluggable Intermediate Data Format
>     
>     Implemented a pluggable intermediate data format that decouples the internal representation
of the data from the connector and the output formats. Connectors can choose to implement
and support a format that is more efficient for them. Also separated the SqoopWritable so
th
>     
>     I ran a full build - all tests including integration tests pass. I have not added
any new tests, yet. I will add unit tests for the new classes. Also, I have not tried running
this on an actual cluster - so things may be broken. I'd like some initial feedback based
on the c
>     
>     I also implemented escaping of characters. There is some work remaining to support
binary format, but it is mostly integration, the basic implementation is in place.
> 
> :100644 100644 db11b5b... 9bfa07d... M  common/pom.xml
> :100644 100644 3e1adc7... f971240... M  common/src/main/java/org/apache/sqoop/etl/io/DataReader.java
> :100644 100644 d81364e... e547875... M  common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java
> :100644 100644 8b630b2... 30c26a3... M  common/src/main/java/org/apache/sqoop/schema/type/Column.java
> :100644 100644 e0da80f... 9c70db9... M  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
> :100644 100644 ef39cdc... 075890f... M  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
> :100644 100644 96818ba... 795ffdb... M  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
> :000000 100644 0000000... 6c33423... A  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/InitializationUtils.java
> :100644 100644 aa1c4ff... 7220018... M  connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
> :100644 100644 a7ed6ba... 28399f2... M  connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
> :100644 100644 4056e14... f54837d... M  connector/connector-sdk/pom.xml
> :000000 100644 0000000... d4874f2... A  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/CSVIntermediateDataFormat.java
> :000000 100644 0000000... 63e14d2... A  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/IntermediateDataFormat.java
> :000000 100644 0000000... 6e5479f... A  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/CSVIntermediateDataFormatTest.java
> :100644 100644 e052584... 3b2ef94... M  core/src/main/java/org/apache/sqoop/framework/JobManager.java
> :100644 100644 53d0039... 9f5c47d... M  core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
> :100644 100644 f9a2a0e... 9754afd... M  execution/mapreduce/pom.xml
> :100644 100644 5c0a027... 7680e33... M  execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
> :100644 100644 7fd9a01... 3604898... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
> :100644 100644 1978ec6... 099cdd3... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java
> :100644 100644 a07c511... ee6bf39... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
> :100644 100644 4621942... 7b799ca... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
> :000000 100644 0000000... 71ae980... A  execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java
> :100644 100644 356ae8a... b495cc9... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java
> :100644 100644 92de37e... 8164ffe... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
> :100644 100644 90de6ef... b3503da... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java
> :100644 100644 7dedee9... 16e59d8... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
> :100644 100644 98a2c51... 5aaceb3... M  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java
> :100644 100644 e21f15b... 4e9e6ea... M  execution/mapreduce/src/test/java/org/apache/sqoop/job/JobUtils.java
> :100644 100644 b7079dd... ddda423... M  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java
> :100644 100644 f849aae... 2968411... M  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java
> :100644 100644 7b264c6... 954990f... M  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java
> :000000 100644 0000000... aea7de3... A  execution/mapreduce/src/test/java/org/apache/sqoop/job/io/SqoopWritableTest.java
> :100644 100644 bee8ab7... 663dfb5... M  execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
> :100644 100644 1e2f005... a722c74... M  pom.xml
> :100644 100644 0b240e8... 43f17d4... M  spi/pom.xml
> :100644 100644 2becc56... 298e8f5... M  spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java
> :100644 100644 6fc485b... 8d1a4e8... M  submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
> 
> 
> Diffs
> -----
> 
>   common/pom.xml db11b5b 
>   common/src/main/java/org/apache/sqoop/etl/io/DataReader.java 3e1adc7 
>   common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java d81364e 
>   common/src/main/java/org/apache/sqoop/schema/type/Column.java 8b630b2 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
e0da80f 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
2b1a0ad 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java
ef39cdc 
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
96818ba 
>   connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java
d4c4565 
>   connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java
a7ed6ba 
>   connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java
a33fa36 
>   connector/connector-sdk/pom.xml 4056e14 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
PRE-CREATION 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java
PRE-CREATION 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormatError.java
PRE-CREATION 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormatTest.java
PRE-CREATION 
>   core/src/main/java/org/apache/sqoop/framework/JobManager.java e052584 
>   core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java a138db5 
>   execution/mapreduce/pom.xml f9a2a0e 
>   execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
5c0a027 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java 7fd9a01 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java
1978ec6 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
a07c511 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
4621942 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java PRE-CREATION

>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java
356ae8a 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 92de37e

>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java
90de6ef 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
7dedee9 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java 98a2c51

>   execution/mapreduce/src/test/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngineTest.java
39d1b53 
>   execution/mapreduce/src/test/java/org/apache/sqoop/job/JobUtils.java e21f15b 
>   execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java b7079dd

>   execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java f849aae 
>   execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 7b264c6 
>   execution/mapreduce/src/test/java/org/apache/sqoop/job/io/SqoopWritableTest.java PRE-CREATION

>   execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java
bee8ab7 
>   pom.xml 1e2f005 
>   spi/pom.xml 0b240e8 
>   spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 2becc56 
>   submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
bfc28ef 
> 
> Diff: https://reviews.apache.org/r/16812/diff/
> 
> 
> Testing
> -------
> 
> mvn test
> 
> 
> Thanks,
> 
> Abraham Elmahrek
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message