hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu Shaohui (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7549) Use JDK ServiceLoader mechanism to find FileSystem implementations
Date Thu, 11 Aug 2016 03:33:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416467#comment-15416467

Liu Shaohui commented on HADOOP-7549:

[~jxchen] [~tucu00]
I encountered same problem when calling CompressionCodecFactory from C++ code through JNI.
java.util.ServiceConfigurationError: org.apache.hadoop.io.compress.CompressionCodec: Provider
org.apache.hadoop.io.compress.BZip2Codec not found
        at java.util.ServiceLoader.fail(ServiceLoader.java:231)
        at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)

The reason is that calling vm->AttachCurrentThread() in jni_helper.c of libhdfs sets the
context class loader to the “bootstrap loader”, which is insufficient for ServiceLoader
to function properly. 
In current implements of libhdfs, the first calling getJNIEnv will create a java vm which
doesn't change the context class loader. But later threads calling getJNIEnv will attach thread
to the java vm which will change the context class loader.  So It works when call FileSystem.loadFileSystems()
in the same thread where JVM is initialized.

A suggest from John Lilley is to setting the system class loader to be the thread's context
class loader:
ps: You must translate it to the JNI callings.
See : [Mail link | http://mail-archives.apache.org/mod_mbox/hadoop-user/201409.mbox/%3Cfc3d75136d434956ae9cc926213576f2@DM2PR0701MB729.namprd07.prod.outlook.com%3E]

> Use JDK ServiceLoader mechanism to find FileSystem implementations
> ------------------------------------------------------------------
>                 Key: HADOOP-7549
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7549
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 2.0.0-alpha
>         Attachments: HADOOP-7549v1.patch, HADOOP-7549v2.patch, HADOOP-7549v3.patch
> Currently configuring FileSystem implementations must be done by declaring the FileSystem
class in the Hadoop configuration files (core-default.xml, ...).
> Using JDK ServiceLoader mechanism this configuration step can be avoided. Adding the
JAR file with the additional FileSystem implementation would suffice. 
> This is similar to what is being proposed for compression codecs (HADOOP-7350).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message