hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11127) Improve versioning and compatibility support in native library for downstream hadoop-common users.
Date Wed, 24 Sep 2014 23:14:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147059#comment-14147059

Chris Nauroth commented on HADOOP-11127:

This initially came up during discussion of HADOOP-11064.  It would be good for anyone interested
to catch up on the full discussion there.  This is a summary of where we got in the discussion
before switching gears in that jira to focus on a quick interim fix.

After further discussion, the following is a summary of potential solutions to this problem:
1. Freeze the libhadoop.so API forever.
2. Library versioning plus maintaining a library on the servers for each supported release.
3. Bundle the .dll or .so file inside a jar somehow so that YARN / Slider can distribute it.

#1. Advantages:
* ???

#1. Disadvantages:
* We're unable to improve libhadoop.so in the future.
* There will be puzzling interactions when mixing and matching versions. "New" bugs in libhadoop.so
will show up with old hadoop releases, causing confusion in bug trackers.
We don't have any way of enforcing C API stability. Jenkins doesn't check for it, most Java
programmers don't know how to achieve it.
* There is still no ability for applications using new Hadoop versions to make use of old
libhadoop.so versions, unless we adopt an even worse compatibility policy that nothing new
can be added to libhadoop.so.
* Given all of the above, this option seems to be off the table.

#2. Advantages:
* Simple to implement.
* There's already a patch that implements it.
* We want libhadoop.so library versioning anyway, even if we later adopt another solution
in addition to this

#2. Disadvantages:
* Admins using Slider / YARN will need to ensure that the appropriate versions of libhadoop
are present on the server.

#3. Advantages:
* "Cleanest" solution, since it allows us to reuse YARN's existing distribution mechanisms.

#3. Disadvantages:
* There are technical challenges to bundling a library in a jar that we haven't yet tackled.

> Improve versioning and compatibility support in native library for downstream hadoop-common
> --------------------------------------------------------------------------------------------------
>                 Key: HADOOP-11127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11127
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>            Reporter: Chris Nauroth
> There is no compatibility policy enforced on the JNI function signatures implemented
in the native library.  This library typically is deployed to all nodes in a cluster, built
from a specific source code version.  However, downstream applications that want to run in
that cluster might choose to bundle a hadoop-common jar at a different version.  Since there
is no compatibility policy, this can cause link errors at runtime when the native function
signatures expected by hadoop-common.jar do not exist in libhadoop.so/hadoop.dll.

This message was sent by Atlassian JIRA

View raw message