spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Schade (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-12453) Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version
Date Mon, 21 Dec 2015 21:56:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067125#comment-15067125
] 

Martin Schade commented on SPARK-12453:
---------------------------------------

Make sense, thank you. Ideally it should be 1.9.37 instead of 1.9.40 though. Both KCL 1.4.0
and KPL 0.10.1 reference 1.9.37.

https://github.com/awslabs/amazon-kinesis-producer/blob/v0.10.1/java/amazon-kinesis-producer/pom.xml

https://github.com/awslabs/amazon-kinesis-client/blob/v1.4.0/pom.xml
Both reference 1.9.37

In the latest version of KPL (v0.10.2) 1.10.34 is references and in latest KCL 1.6.1 it is
version 1.10.20, not easy so sync then. So it would need to do some testing which combination
works actually.

> Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version
> ------------------------------------------------------------------------
>
>                 Key: SPARK-12453
>                 URL: https://issues.apache.org/jira/browse/SPARK-12453
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.5.2
>            Reporter: Martin Schade
>            Priority: Critical
>              Labels: easyfix
>
> The Spark Streaming Kinesis Example (kinesis-asl) is broken due to wrong AWS Java SDK
version (1.9.16) referenced with the AWS KCL version (1.3.0).
> AWS KCL 1.3.0 references AWS Java SDK version 1.9.37.
> Using 1.9.16 in combination with 1.3.0 does fail to get data out of the stream.
> I tested Spark Streaming with 1.9.37 and it works fine. 
> Testing a simple KCL client outside of Spark with 1.3.0 and 1.9.16 also fails, so it
is due to the specific versions used in 1.5.2 and not a Spark related implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message