hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13060) While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
Date Tue, 26 Apr 2016 08:54:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257768#comment-15257768
] 

Steve Loughran commented on HADOOP-13060:
-----------------------------------------

Jenny: you need to have the AWS versions of your libraries in sync with what Hadoop was built
against; amazon changed the signature of a method, moving an argument from an int to a long,
which is something that ends up getting frozen at compile time. 

If you are working with Hadoop 2.7.x, then you will need aws-java-sdk 1.7.4 on your classpath.
Sorry, but we can't do anything about amazon quietly breaking binary compatibility on their
JARs.

Now, if you really want the latest aws SDK against hadoop  2.7.x, you can actually rebuild
hadoop (more specifically, tools/hadoop-aws) against the latest amazon JAR; it's just you
need to do it at compile time, after which the library choice is frozen

> While trying to perform a Distcp command, we see the error Exception in thread "main"
java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13060
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13060
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.1
>            Reporter: Jenny Dong
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the following
error/stacktrace : 
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar version 1.10.69
(we bumped this up from 2.7.4 because we were getting errors seen in HADOOP-12420 + other
authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> 	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> 	at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
> 	at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
> 	at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
> 	at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & org.apache.hadoop.fs.s3a.S3AFileSystem.
The only difference is S3AFileSystem created a ThreadPoolExecutor (which implements AbstractExecutorService
which implements ExecutorService). I also checked on the classpath to make sure the version
of the jars being picked up is what I expected. 
> Help would be much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message