spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5348) s3a:// protocol and hadoop-aws dependency
Date Thu, 22 Jan 2015 10:45:36 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287243#comment-14287243
] 

Sean Owen commented on SPARK-5348:
----------------------------------

Hm, but wouldn't Spark have the same problem distributing this library then?

You can just bundle it with your app, right? or would the Hadoop mechanism not find the handlers
for {{s3a://}} in that case?
If not then isn't the solution to deploy this library with your Hadoop 2.6 installation?

> s3a:// protocol and hadoop-aws dependency
> -----------------------------------------
>
>                 Key: SPARK-5348
>                 URL: https://issues.apache.org/jira/browse/SPARK-5348
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 1.2.0
>            Reporter: Rafal Kwasny
>            Priority: Minor
>
> Hadoop 2.6.0+ includes new s3a library which is a lot better than the old one.
> https://issues.apache.org/jira/browse/HADOOP-10400
> Unfortunately this library is linking with amazon sdk which I think is not compatible
with apache licensing. For this reason it is not a dependency for hadoop-client.
> What I did is just add a profile for maven build, not sure if there is a better option
to integrate this, we could also add a new hadoop-2.6 profile.
> {noformat}
> diff --git a/pom.xml b/pom.xml
> index a9e7c9d..71ee776 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -1214,6 +1214,27 @@
>      </profile>
>      <profile>
> +      <id>hadoop-aws</id>
> +      <dependencies>
> +        <dependency>
> +          <groupId>org.apache.hadoop</groupId>
> +          <artifactId>hadoop-aws</artifactId>
> +          <version>${hadoop.version}</version>
> +          <exclusions>
> +            <exclusion>
> +              <groupId>com.fasterxml.jackson.core</groupId>
> +              <artifactId>jackson-annotations</artifactId>
> +            </exclusion>
> +            <exclusion>
> +              <groupId>com.fasterxml.jackson.core</groupId>
> +              <artifactId>jackson-databind</artifactId>
> +            </exclusion>
> +          </exclusions>
> +        </dependency>
> +      </dependencies>
> +    </profile>
> +
> +    <profile>
>        <id>java8-tests</id>
>        <build>
>          <plugins>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message