hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
Date Wed, 10 Sep 2014 01:12:29 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127882#comment-14127882

Colin Patrick McCabe commented on HADOOP-11064:

bq. I think that's a good sign that we could port hadoop.dll to use CMake and eliminate these
kinds of pitfalls. That's a topic for another time though.

I agree, it would be great to see the Windows build of {{hadoop.dll}} use CMake.

bq. FWIW, I'd prefer some kind of solution that bundles the native build into a versioned
jar. As far as I can tell, that provides the cleanest isolation for this use case. Unfortunately,
the build infrastructure requirements for this look infeasible to me in the short-term.

It seems like this is a general issue with native dependencies of jars submitted through YARN.
 It would be better to solve it inside YARN, than to force a draconian internal API compatibility
policy on to libhadoop.

bq. These issues make me think the versioning solution needs additional design work and buy-in
from a large portion of the community before we settle on it. Restoring the old function signatures
would be helpful to unblock downstream projects that are trying to test against 2.6.0-SNAPSHOT
clusters right now. Granted, 2.6.0-SNAPSHOT isn't a real release, so we're technically under
no obligation, but I see it as a matter of good citizenship.

I think the versioning solution does show good citizenship.  We are going out of our way to
work around this issue with YARN and native dependencies.  It's not a perfect workaround,
but I don't see how it can be, unless YARN is changed to somehow distribute these dependencies
(as you hinted at?)

Also, what are the "old" function signatures?  The ones for Hadoop 2.4?  2.5?  2.5.1?  libhadoop
changes all the time and its development isn't frozen.

I am + 1 for v3 of the patch... and based on previous conversations, I think most people would
be OK with adding versioning to libhadoop.  Perhaps bring it up on hadoop-dev if you think
this needs more eyes?

I apologize again for being difficult here... you guys have done a lot of good work on the
native library and on Windows support.  But I want to make sure we're not constrained in the
future.  It may be a while before we can get a perfect solution to this.

> UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-11064
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11064
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.6.0
>         Environment: Hadoop 2.6 cluster, trying to run code containing hadoop 2.4 JARs
>            Reporter: Steve Loughran
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>         Attachments: HADOOP-11064.001.patch, HADOOP-11064.002.patch, HADOOP-11064.003.patch
> The private native method names and signatures in {{NativeCrc32}} were changed in HDFS-6561
... as a result hadoop-common-2.4 JARs get unsatisifed link errors when they try to perform
> This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless rebuilt and
repackaged with the hadoop- 2.6 JARs

This message was sent by Atlassian JIRA

View raw message