sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Szabolcs Vasas <vasas.szabo...@gmail.com>
Subject Re: Review Request 67628: Implement an alternative solution for Parquet reading and writing
Date Mon, 18 Jun 2018 09:49:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67628/
-----------------------------------------------------------

(Updated June 18, 2018, 9:49 a.m.)


Review request for Sqoop.


Bugs: SQOOP-3328
    https://issues.apache.org/jira/browse/SQOOP-3328


Repository: sqoop-trunk


Description
-------

The new implementation uses classes from parquet.hadoop packages.
TestParquetIncrementalImportMerge has been introduced to cover some gaps we had in the Parquet
merge support.
The test infrastructure is also modified a bit which was needed because of TestParquetIncrementalImportMerge.

Note that this JIRA does not cover the Hive Parquet import support I will create another JIRA
for that.


Diffs
-----

  src/java/org/apache/sqoop/SqoopOptions.java d9984af36 
  src/java/org/apache/sqoop/avro/AvroUtil.java 57c206256 
  src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 3b5421028 
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 17c9ed39b 
  src/java/org/apache/sqoop/mapreduce/parquet/ParquetJobConfiguratorFactoryProvider.java 2286a5203

  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopMergeParquetReducer.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportMapper.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportMapper.java PRE-CREATION

  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetJobConfiguratorFactory.java
PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
PRE-CREATION 
  src/test/org/apache/sqoop/TestBigDecimalExport.java ccea17345 
  src/test/org/apache/sqoop/TestMerge.java 11806fea6 
  src/test/org/apache/sqoop/TestParquetExport.java 43dabb57b 
  src/test/org/apache/sqoop/TestParquetImport.java 27d407aa3 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java PRE-CREATION 
  src/test/org/apache/sqoop/hive/TestHiveImport.java 436f0e512 
  src/test/org/apache/sqoop/hive/TestHiveServer2TextImport.java f6d591b73 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerHiveImportTest.java e6b086550 
  src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java a5f85a06b 
  src/test/org/apache/sqoop/testutil/ImportJobTestCase.java dbefe2097 
  src/test/org/apache/sqoop/util/ParquetReader.java 56e03a060 


Diff: https://reviews.apache.org/r/67628/diff/1/


Testing (updated)
-------

Ran unit and third party tests successfully.


Thanks,

Szabolcs Vasas


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message