hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11263) NativeS3FileSystem doesn't work with hadoop-client
Date Tue, 04 Nov 2014 09:41:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195959#comment-14195959

Steve Loughran commented on HADOOP-11263:

It's  not in there to keep the dependency graph down; add jets3t and will work

All S3 support moves in 2.6+ to the hadoop-aws JAR, along with s3a:; these do explicitly add
the dependencies, on the basis that if you pull in that JAR, you want the dependencies,

I'm closing this as a wontfix. Sorry

> NativeS3FileSystem doesn't work with hadoop-client
> --------------------------------------------------
>                 Key: HADOOP-11263
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11263
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Sangjin Lee
> When you start using the NativeS3FileSystem (which is in hadoop-common) based on the
hadoop-client set of jars, it fails with a NoClassDefFoundError:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException
> 	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:280)
> 	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:270)
> 	at com.twitter.twadoop.util.hadoop.NativeS3FileSystemWrapper.initialize(NativeS3FileSystemWrapper.java:34)
> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2438)
> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
> 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2472)
> 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2454)
> 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:384)
> 	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> {noformat}
> NativeS3FileSystem depends on a library called jets3t, which is not found in the hadoop-client
build. It turns out that this library was specifically excluded in the hadoop-client pom.xml:
> {noformat}
>         <exclusion>
>           <groupId>net.java.dev.jets3t</groupId>
>           <artifactId>jets3t</artifactId>
>         </exclusion>
> {noformat}
> This strikes me as an issue, as a component that's part of hadoop-common cannot run with
a hadoop-client build.

This message was sent by Atlassian JIRA

View raw message