sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fero Szabo via Review Board <nore...@reviews.apache.org>
Subject Re: Review Request 69060: SQOOP-3382 Add parquet numeric support for Parquet in hdfs import
Date Thu, 18 Oct 2018 17:40:07 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69060/
-----------------------------------------------------------

(Updated Oct. 18, 2018, 5:40 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Changes
-------

Added a new flag for enabling logical types with parquet. This should allow backward compatibility.


Summary (updated)
-----------------

SQOOP-3382 Add parquet numeric support for Parquet in hdfs import


Bugs: SQOOP-3382
    https://issues.apache.org/jira/browse/SQOOP-3382


Repository: sqoop-trunk


Description
-------

This patch is about adding support for fixed point decimal types in parquet import.

The implementation is simple after the fact that parquet was upgraded to 1.9.0 in SQOOP-3381:
we just need to register the GenericDataSupplier with AvroParquetOutputFormat.

For testing, we can reuse the existing Avro tests, because Sqoop uses Avro under the hood
to write parquet.

I also moved around and renamed the classes involved in this change so their name and package
reflect their purpose.


Diffs (updated)
-----

  src/java/org/apache/sqoop/config/ConfigurationConstants.java 3724f250 
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 80c06988 
  src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java 62334f8a 
  src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 7a2a5f9c 
  src/test/org/apache/sqoop/importjob/ImportJobTestConfiguration.java 14de910b 
  src/test/org/apache/sqoop/importjob/avro/AvroImportForNumericTypesTest.java ff13dc3b 
  src/test/org/apache/sqoop/importjob/avro/configuration/MSSQLServerImportJobTestConfiguration.java
182d2967 
  src/test/org/apache/sqoop/importjob/avro/configuration/MySQLImportJobTestConfiguration.java
e9bf9912 
  src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfiguration.java
b7bad08c 
  src/test/org/apache/sqoop/importjob/avro/configuration/OracleImportJobTestConfigurationForNumber.java
465e61f4 
  src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationForNumeric.java
66715c17 
  src/test/org/apache/sqoop/importjob/avro/configuration/PostgresqlImportJobTestConfigurationPaddingShouldSucceed.java
ec4db41b 
  src/test/org/apache/sqoop/util/ParquetReader.java 908ce566 


Diff: https://reviews.apache.org/r/69060/diff/2/

Changes: https://reviews.apache.org/r/69060/diff/1-2/


Testing
-------

3rd party tests and unit tests, both gradle and ant


Thanks,

Fero Szabo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message