datafu-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Hayes" <>
Subject Review Request 34636: DATAFU-58 Update to Hadoop 2.7.0 and Pig 0.14.0 and fix all tests
Date Sat, 23 May 2015 20:15:36 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for DataFu.

Repository: datafu


This updates DataFu to use Hadoop 2.7.0 and Pig 0.14.0.  I've fixed all issues that I could
find.  All the unit tests pass.  Some of the issues were already addressd by Daniel Dai's
earlier patch, but I found some more problems.

Summary of issues:

* Hourglass tests failed because we were pulling in avro-tools, which contains Hadoop classes
and therefore conflicts.  I removed the dependency.  I'm not sure why we had this.  I also
added a handy script to help with this analysis, which uses jarfish.
* Hourglass used .toString() for many Path instances.  This caused a problem because the resulting
string is prefixed with "file:" and therefore isn't a valid path you can use in File.  The
general fix is to use getName() instead, which gets the simple name and is usually suitable.
* Some Pig unit tests had assumptions on tuple ordering in bags.  I applied a sort to the
actual and expected bags to ensure a consistent comparison.

----- 8e1b67d 
  build-plugin/src/main/java/org/adrianwalker/multilinestring/ 9abdba5

  datafu-hourglass/.gitignore 942515e 
  datafu-hourglass/build.gradle 75a2876 
  datafu-hourglass/find_dupes.rb PRE-CREATION 
  datafu-hourglass/src/main/java/datafu/hourglass/fs/ c270c7b 
  datafu-hourglass/src/main/java/datafu/hourglass/mapreduce/ 62975d1

  datafu-hourglass/src/test/java/datafu/hourglass/demo/ 039822c 
  datafu-hourglass/src/test/java/datafu/hourglass/test/ 02aa342

  datafu-hourglass/src/test/java/datafu/hourglass/test/ fff1cfd

  datafu-hourglass/src/test/java/datafu/hourglass/test/ c41fd39

  datafu-hourglass/src/test/java/datafu/hourglass/test/ acae96c

  datafu-hourglass/src/test/java/datafu/hourglass/test/ b428003 
  datafu-hourglass/src/test/java/datafu/hourglass/test/ bc52977 
  datafu-pig/build.gradle ea385d2 
  datafu-pig/src/test/java/datafu/test/pig/bags/ 9bcc384 
  gradle/dependency-versions.gradle 3b0835f 



./gradlew test


Matthew Hayes

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message