sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daniel voros <daniel.vo...@gmail.com>
Subject Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library
Date Tue, 16 Oct 2018 15:12:00 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/#review209630
-----------------------------------------------------------


Ship it!




Thanks Fero, for taking care of this. Ship it!

- daniel voros


On Oct. 16, 2018, 9:37 a.m., Fero Szabo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68687/
> -----------------------------------------------------------
> 
> (Updated Oct. 16, 2018, 9:37 a.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3381
>     https://issues.apache.org/jira/browse/SQOOP-3381
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> This change upgrades our parquet library to the newest version and a whole lot of libraries
to newer versions with it.
> 
> As we will need to register a data supplier in the fix for parquet decimal support (SQOOP-3382),
we will need a version that contains PARQUET-243. We need to upgrade the Parquet library to
a version that contains this fix and is compatible with Hadoop 3.0.
> 
> A few things to note:
> - hadoop's version is still 2.8.0
> - hive is upgraded to 2.1.1
> - the rest of the dependency changes are required for the hive version bump.
> 
> There is are a few changes in the codebase, but of course no new functionality at all:
> - in the TestParquetImport class, the new implementation returns a Utf8 object for Strings
written out.
> - Added the security policy and related code changes from the patch for SQOOP-3305 (upgrade
hadoop) written by Daniel Voros.
> - modified HiveMiniCluster config so it won't try to start a web ui (it's unnecessary
during tests anyway)
> 
> 
> Diffs
> -----
> 
>   build.gradle fc7fc0c4 
>   gradle.properties 0d30378d 
>   gradle/sqoop-package.gradle 1a8d994d 
>   ivy.xml 670cb32d 
>   ivy/libraries.properties 8f3dab2b 
>   src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
>   src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
>   src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
2180cc20 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
90b910a3 
>   src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
66ebc5b8 
>   src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
>   src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
>   src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
>   src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
>   src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
>   testdata/hcatalog/conf/hive-site.xml 8a84a5d3 
> 
> 
> Diff: https://reviews.apache.org/r/68687/diff/5/
> 
> 
> Testing
> -------
> 
> Ant unit and 3rd party tests were successful.
> gradlew test and thirdpartytest were succesful as well.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message