sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1359) Fix avro versions in Sqoop to stop shipping hadoop1 jars with hadoop2
Date Mon, 21 Jul 2014 14:18:38 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068554#comment-14068554

Venkat Ranganathan commented on SQOOP-1359:

This is getting tricky from the build point of view.  Some projects do not use hadoop2/hadoop1
classifier (and are not dependent on avro-mapred.jar hadoop1/hadoop2 specifiic classes). 
 So it is more likely that you have in your ivy and m2 cache, the avro-mapred.jar cached.
   I added the classifier for hadoop2 builds, but the build fails when the cached artifact
in the local ivy and m2 caches do not have the right classifier.   We have to clean the avro-mapred
cached jars in m2 and ivy caches to make the sqoop build successful and this may affect building
other components.  I tried to update ivy to the latest release version 2.3.0 but still ivy
does not download the jar with hadoop2 classifier if it finds something without the classifier
in the m2 cache.

Looking at Sqoop, also we don't really depend on avro-mapred hadoop1/hadoop2 dependent classes
(those that deal with TaskAttamptContexts).   

I think we have two options:
Just update avro version alone without specifying classifiers (which avro version would be
question - 1.7.4 is used by hadoop, and 1.7.5 is used by hive and they are some incompatibilities
between the versions).  

Or recommend users to nuke the avro-mapred cached jars before building sqoop and any other
avro-mapred dependent builds.


> Fix avro versions in Sqoop to stop shipping hadoop1  jars with hadoop2
> ----------------------------------------------------------------------
>                 Key: SQOOP-1359
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1359
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.4, 1.4.5
>            Reporter: Venkat Ranganathan
>            Assignee: Venkat Ranganathan
>             Fix For: 1.4.5
> AVRO-1170 fixed avro-mapred for hadoop2 and we now have avro-mapred hadoop1 and hadoop2
specific jars.   Our dependency should be updated to use the right jars.   I recently found
similar issues with Hive shipping avro-mapred hadoop1 jars (HIVE-7240) causing issues in certain
integration scenarios.

This message was sent by Atlassian JIRA

View raw message