datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Hayes" <matthew.terence.ha...@gmail.com>
Subject Review Request 34636: DATAFU-58 Update to Hadoop 2.7.0 and Pig 0.14.0 and fix all tests
Date Sat, 23 May 2015 20:15:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34636/
-----------------------------------------------------------

Review request for DataFu.


Repository: datafu


Description
-------

This updates DataFu to use Hadoop 2.7.0 and Pig 0.14.0.  I've fixed all issues that I could
find.  All the unit tests pass.  Some of the issues were already addressd by Daniel Dai's
earlier patch, but I found some more problems.

Summary of issues:

* Hourglass tests failed because we were pulling in avro-tools, which contains Hadoop classes
and therefore conflicts.  I removed the dependency.  I'm not sure why we had this.  I also
added a handy script to help with this analysis, which uses jarfish.
* Hourglass used .toString() for many Path instances.  This caused a problem because the resulting
string is prefixed with "file:" and therefore isn't a valid path you can use in File.  The
general fix is to use getName() instead, which gets the simple name and is usually suitable.
* Some Pig unit tests had assumptions on tuple ordering in bags.  I applied a sort to the
actual and expected bags to ensure a consistent comparison.


Diffs
-----

  README.md 8e1b67d 
  build-plugin/src/main/java/org/adrianwalker/multilinestring/MultilineProcessor.java 9abdba5

  datafu-hourglass/.gitignore 942515e 
  datafu-hourglass/build.gradle 75a2876 
  datafu-hourglass/find_dupes.rb PRE-CREATION 
  datafu-hourglass/src/main/java/datafu/hourglass/fs/PathUtils.java c270c7b 
  datafu-hourglass/src/main/java/datafu/hourglass/mapreduce/DistributedCacheHelper.java 62975d1

  datafu-hourglass/src/test/java/datafu/hourglass/demo/Examples.java 039822c 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingExecutionPlannerTests.java
d68ea83 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingJoinTest.java 02aa342

  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingTests.java fff1cfd

  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingCollapsingIntegrationTests.java
a8f020b 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingJoinTests.java c41fd39

  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingTests.java acae96c

  datafu-hourglass/src/test/java/datafu/hourglass/test/TestAvroJob.java b428003 
  datafu-hourglass/src/test/java/datafu/hourglass/test/TestBase.java bc52977 
  datafu-pig/build.gradle ea385d2 
  datafu-pig/src/test/java/datafu/test/pig/bags/BagTests.java 9bcc384 
  gradle/dependency-versions.gradle 3b0835f 

Diff: https://reviews.apache.org/r/34636/diff/


Testing
-------

./gradlew test


Thanks,

Matthew Hayes


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message