sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abraham Elmahrek" <...@cloudera.com>
Subject Re: Review Request 24878: SQOOP-1375 - HDFS connector
Date Wed, 20 Aug 2014 05:46:22 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24878/#review51070
-----------------------------------------------------------


Good stuff Gwen! A few comments below, but in general +1 from me!


common/pom.xml
<https://reviews.apache.org/r/24878/#comment89021>

    This seems similar to the parent pom definition. You should be able to remove the activation
configuration since it exists in the parent pom.



common/pom.xml
<https://reviews.apache.org/r/24878/#comment89022>

    This seems similar to the parent pom definition. You should be able to remove the activation
configuration since it exists in the parent pom.



common/pom.xml
<https://reviews.apache.org/r/24878/#comment89023>

    Not necessary here?



connector/connector-generic-hdfs/pom.xml
<https://reviews.apache.org/r/24878/#comment89025>

    sqoop-connector-hdfs?



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsConnector.java
<https://reviews.apache.org/r/24878/#comment89026>

    Should never hit this case... but I feel like we should throw an exception.



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsDestroyer.java
<https://reviews.apache.org/r/24878/#comment89027>

    Follow up Jira for sure! A "done" flag is super useful for integration with Oozie.



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsInitializer.java
<https://reviews.apache.org/r/24878/#comment89029>

    Follow up Jira to define some kind of Schema? Can expect CSV data and fields are Strings
at the minimum. The framework should be able to handle coercing types!



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsLoader.java
<https://reviews.apache.org/r/24878/#comment89031>

    rm -rf?



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsLoader.java
<https://reviews.apache.org/r/24878/#comment89030>

    rm -rf



connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ConnectionForm.java
<https://reviews.apache.org/r/24878/#comment89032>

    Follow up Jira:
    
    Hostname, port, scheme, or some mixture of every thing? For example:
    
    hdfs://notmylocalhost:8050



connector/pom.xml
<https://reviews.apache.org/r/24878/#comment89024>

    Yes!



core/src/main/java/org/apache/sqoop/framework/FrameworkValidator.java
<https://reviews.apache.org/r/24878/#comment89013>

    Nit: Kill off some of the whitespace here.



core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
<https://reviews.apache.org/r/24878/#comment89015>

    Nit: Spacing seems weird here.



core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
<https://reviews.apache.org/r/24878/#comment89016>

    Nit: Kill space.



execution/mapreduce/src/test/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngineTest.java
<https://reviews.apache.org/r/24878/#comment89017>

    Nit: Tab oddness.



server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java
<https://reviews.apache.org/r/24878/#comment89018>

    Nit: Spacing.


- Abraham Elmahrek


On Aug. 20, 2014, 2:03 a.m., Gwen Shapira wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24878/
> -----------------------------------------------------------
> 
> (Updated Aug. 20, 2014, 2:03 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-SQOOP-1367
> 
> 
> Description
> -------
> 
> Refactored framework code into an HDFS connector
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/sqoop/client/SqoopClient.java 1d93ae3 
>   common/pom.xml 9bfa07d 
>   common/src/main/java/org/apache/sqoop/common/PrefixContext.java PRE-CREATION 
>   connector/connector-generic-hdfs/pom.xml PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsConnector.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsConnectorError.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsConstants.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsDestroyer.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsExtractor.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsInitializer.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsLoader.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsPartition.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsPartitioner.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/GenericHdfsValidator.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ConnectionConfiguration.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ConnectionForm.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/FromJobConfiguration.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/InputForm.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/OutputCompression.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/OutputForm.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/OutputFormat.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/StorageType.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/configuration/ToJobConfiguration.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsSequenceWriter.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/HdfsTextWriter.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/hdfsWriter/genericHdfsWriter.java
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/resources/generic-hdfs-connector-resources.properties
PRE-CREATION 
>   connector/connector-generic-hdfs/src/main/resources/sqoopconnector.properties PRE-CREATION

>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
39d48c7 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormatError.java
9219074 
>   connector/pom.xml d642c3e 
>   core/src/main/java/org/apache/sqoop/framework/FrameworkValidator.java f19a23e 
>   core/src/main/java/org/apache/sqoop/framework/JobManager.java 48360ea 
>   core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java be6099e 
>   core/src/main/java/org/apache/sqoop/framework/configuration/ExportJobConfiguration.java
6665429 
>   core/src/main/java/org/apache/sqoop/framework/configuration/ImportJobConfiguration.java
2a35eb9 
>   core/src/main/java/org/apache/sqoop/framework/configuration/InputForm.java d5cbeec

>   core/src/main/java/org/apache/sqoop/framework/configuration/JobConfiguration.java 7c653bf

>   core/src/main/java/org/apache/sqoop/framework/configuration/OutputCompression.java
6cac46d 
>   core/src/main/java/org/apache/sqoop/framework/configuration/OutputForm.java b2cdb44

>   core/src/main/java/org/apache/sqoop/framework/configuration/OutputFormat.java 4cd3589

>   core/src/main/java/org/apache/sqoop/framework/configuration/StorageType.java dbe9f95

>   core/src/test/java/org/apache/sqoop/framework/TestFrameworkValidator.java f875ceb 
>   core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java 31df04c 
>   execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
ff328cb 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/PrefixContext.java c3beed7 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java
27afd8c 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartition.java
cdbdaa8 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportPartitioner.java
b3590dc 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java
d4ffb13 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
7b799ca 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
8e31ef5 
>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java e96909a

>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 6e2cfbf

>   execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
01c32e4 
>   execution/mapreduce/src/test/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngineTest.java
0f2a882 
>   execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java e460c3e

>   execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java 65e82b1 
>   pom.xml 38b4974 
>   repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
f355ceb 
>   server/pom.xml dc89409 
>   server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java 6f2df93 
>   test/pom.xml 8001fce 
>   test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 5ec4fa4 
>   test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/TableImportTest.java
9e6f991 
>   test/src/test/java/org/apache/sqoop/integration/connector/jdbc/generic/imports/PartitionerTest.java
1bc3b93 
>   test/src/test/java/org/apache/sqoop/integration/server/SubmissionWithDisabledModelObjectsTest.java
126ca32 
> 
> Diff: https://reviews.apache.org/r/24878/diff/
> 
> 
> Testing
> -------
> 
> From MySQL to HDFS uncompressed text
> and from HDFS uncompressed text to MySQL
> 
> 
> Thanks,
> 
> Gwen Shapira
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message