hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-17337) S3A NetworkBinding has a runtime class dependency on a third-party shaded class
Date Tue, 01 Jun 2021 16:51:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-17337:
------------------------------------
        Parent: HADOOP-17566
    Issue Type: Sub-task  (was: Bug)

> S3A NetworkBinding has a runtime class dependency on a third-party shaded class
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-17337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17337
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Chris Wensel
>            Assignee: Steve Loughran
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 3.3.1
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which
in turn is a fat jar of all AWS SDK libraries and shaded dependencies.
>  
> This dependency is 181MB.
>  
> Some applications using the S3AFilesystem may be sensitive to having a large footprint.
For example, building an application using Parquet and bundled with Docker.
>  
> Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK
dependencies, dropping the overall footprint.
>  
> In 3.3 (and maybe prior versions) this strategy does not work because of the following
exception: 
> {{java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
> {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
> {{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
> {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
> {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
> {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
> {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
> {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
> {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message