[ https://issues.apache.org/jira/browse/SPARK-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287243#comment-14287243
]
Sean Owen commented on SPARK-5348:
----------------------------------
Hm, but wouldn't Spark have the same problem distributing this library then?
You can just bundle it with your app, right? or would the Hadoop mechanism not find the handlers
for {{s3a://}} in that case?
If not then isn't the solution to deploy this library with your Hadoop 2.6 installation?
> s3a:// protocol and hadoop-aws dependency
> -----------------------------------------
>
> Key: SPARK-5348
> URL: https://issues.apache.org/jira/browse/SPARK-5348
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.2.0
> Reporter: Rafal Kwasny
> Priority: Minor
>
> Hadoop 2.6.0+ includes new s3a library which is a lot better than the old one.
> https://issues.apache.org/jira/browse/HADOOP-10400
> Unfortunately this library is linking with amazon sdk which I think is not compatible
with apache licensing. For this reason it is not a dependency for hadoop-client.
> What I did is just add a profile for maven build, not sure if there is a better option
to integrate this, we could also add a new hadoop-2.6 profile.
> {noformat}
> diff --git a/pom.xml b/pom.xml
> index a9e7c9d..71ee776 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -1214,6 +1214,27 @@
> </profile>
> <profile>
> + <id>hadoop-aws</id>
> + <dependencies>
> + <dependency>
> + <groupId>org.apache.hadoop</groupId>
> + <artifactId>hadoop-aws</artifactId>
> + <version>${hadoop.version}</version>
> + <exclusions>
> + <exclusion>
> + <groupId>com.fasterxml.jackson.core</groupId>
> + <artifactId>jackson-annotations</artifactId>
> + </exclusion>
> + <exclusion>
> + <groupId>com.fasterxml.jackson.core</groupId>
> + <artifactId>jackson-databind</artifactId>
> + </exclusion>
> + </exclusions>
> + </dependency>
> + </dependencies>
> + </profile>
> +
> + <profile>
> <id>java8-tests</id>
> <build>
> <plugins>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|