hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11064) UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
Date Tue, 09 Sep 2014 21:14:30 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HADOOP-11064:
    Attachment: HADOOP-11064.002.patch

The hadoop.dll build is still driven from a checked-in project file instead of CMake.  Because
of this, certain build changes require duplication of logic for both CMake and MSBuild.  I've
attached a v2 patch showing the change.  This just covers hadoop.dll.  We'll also need to
consider winutils.exe, so the patch isn't complete yet.  (BTW for anyone observing, I used
CMake on Windows for the libhdfs port in HDFS-573, so I think that's a good sign that we could
port hadoop.dll to use CMake and eliminate these kinds of pitfalls.  That's a topic for another
time though.)

bq. I think the versioning solution is quite clean and easy to implement.

I agree, but I also think there are still some unresolved issues driven by the use case Steve

The basic use case is to submit a YARN application with complete control of the Hadoop dependencies.
 The proposed patch would publish versioned builds of the form libhadoop-<version>.so.
 The {{System.loadLibrary}} call would match strictly on the exact jar version.

However, the application owner might decide to upgrade the project's Hadoop dependency before
the cluster administrator deploys the new libhadoop-<version>.so, which would result
in the application unexpectedly getting link errors.  Note that this would be a problem even
for minor upgrades, where we've stated a guarantee of backwards-compatibility.  If the application
changes its dependency from 2.4.1 to 2.4.2, even if we didn't change any native code in that
rev, the application will get a link error due to missing libhadoop-2.4.2.so on the cluster.

This then implies that cluster administrators have a responsibility to deploy and maintain
copies of every new point release of every major release ever deployed, just in case one of
the devs they support decides to submit an application with that version.  Things that used
to be routine deployments now might require more explicit coordination between application
developers and cluster administrators.  Most administrators would prefer to deploy only the
software versions that they're really interested in running.

FWIW, I'd prefer some kind of solution that bundles the native build into a versioned jar.
 As far as I can tell, that provides the cleanest isolation for this use case.  Unfortunately,
the build infrastructure requirements for this look infeasible to me in the short-term.

These issues make me think the versioning solution needs additional design work and buy-in
from a large portion of the community before we settle on it.  Restoring the old function
signatures would be helpful to unblock downstream projects that are trying to test against
2.6.0-SNAPSHOT clusters right now.  Granted, 2.6.0-SNAPSHOT isn't a real release, so we're
technically under no obligation, but I see it as a matter of good citizenship.

> UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due NativeCRC32 method changes
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-11064
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11064
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 2.6.0
>         Environment: Hadoop 2.6 cluster, trying to run code containing hadoop 2.4 JARs
>            Reporter: Steve Loughran
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>         Attachments: HADOOP-11064.001.patch, HADOOP-11064.002.patch
> The private native method names and signatures in {{NativeCrc32}} were changed in HDFS-6561
... as a result hadoop-common-2.4 JARs get unsatisifed link errors when they try to perform
> This essentially stops Hadoop 2.4 applications running on Hadoop 2.6 unless rebuilt and
repackaged with the hadoop- 2.6 JARs

This message was sent by Atlassian JIRA

View raw message