sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fero Szabo via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 67628: Implement an alternative solution for Parquet reading and writing
Date Mon, 18 Jun 2018 13:59:51 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67628/#review204919
-----------------------------------------------------------




src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopMergeParquetReducer.java
Lines 30 (patched)
<https://reviews.apache.org/r/67628/#comment287721>

    



src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java
Line 145 (original), 147 (patched)
<https://reviews.apache.org/r/67628/#comment287732>

    Might be a good idea to remove the Sysout while this is modified. 
    
    Also, you could consider using the new builder pattern, though I see that might be better
suited for a separate Jira.



src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java
Line 147 (original), 149 (patched)
<https://reviews.apache.org/r/67628/#comment287730>

    So, Hadoop flags are no longer included. Was that your intention here? If so, the 'includeHadoopFlags'
boolean parameter is confusing for me.



src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java
Line 147 (original), 149 (patched)
<https://reviews.apache.org/r/67628/#comment287731>

    



src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java
Lines 669 (patched)
<https://reviews.apache.org/r/67628/#comment287729>

    Might not be an issue, but shouldn't this handle dates as well?
    
    I can't think of any other text-like types.
    
    BigDecimals are probably OK.


- Fero Szabo


On June 18, 2018, 9:49 a.m., Szabolcs Vasas wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67628/
> -----------------------------------------------------------
> 
> (Updated June 18, 2018, 9:49 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3328
>     https://issues.apache.org/jira/browse/SQOOP-3328
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> The new implementation uses classes from parquet.hadoop packages.
> TestParquetIncrementalImportMerge has been introduced to cover some gaps we had in the
Parquet merge support.
> The test infrastructure is also modified a bit which was needed because of TestParquetIncrementalImportMerge.
> 
> Note that this JIRA does not cover the Hive Parquet import support I will create another
JIRA for that.
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/SqoopOptions.java d9984af36 
>   src/java/org/apache/sqoop/avro/AvroUtil.java 57c206256 
>   src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 3b5421028 
>   src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 17c9ed39b 
>   src/java/org/apache/sqoop/mapreduce/parquet/ParquetJobConfiguratorFactoryProvider.java
2286a5203 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopMergeParquetReducer.java PRE-CREATION

>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportMapper.java PRE-CREATION

>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportMapper.java PRE-CREATION

>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetJobConfiguratorFactory.java
PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
PRE-CREATION 
>   src/test/org/apache/sqoop/TestBigDecimalExport.java ccea17345 
>   src/test/org/apache/sqoop/TestMerge.java 11806fea6 
>   src/test/org/apache/sqoop/TestParquetExport.java 43dabb57b 
>   src/test/org/apache/sqoop/TestParquetImport.java 27d407aa3 
>   src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java PRE-CREATION 
>   src/test/org/apache/sqoop/hive/TestHiveImport.java 436f0e512 
>   src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java f6d591b73 
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java e6b086550

>   src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java a5f85a06b 
>   src/test/org/apache/sqoop/testutil/ImportJobTestCase.java dbefe2097 
>   src/test/org/apache/sqoop/util/ParquetReader.java 56e03a060 
> 
> 
> Diff: https://reviews.apache.org/r/67628/diff/1/
> 
> 
> Testing
> -------
> 
> Ran unit and third party tests successfully.
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message