spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "A.K.M. Ashrafuzzaman" <ashrafuzzaman...@gmail.com>
Subject Re: Having problem with Spark streaming with Kinesis
Date Sun, 14 Dec 2014 04:35:38 GMT
Thanks Aniket,
The trick is to have the #workers >= #shards + 1. But I don’t know why is that.
http://spark.apache.org/docs/latest/streaming-kinesis-integration.html

Here in the figure[spark streaming kinesis architecture], it seems like one node should be
able to take on more than one shards.


A.K.M. Ashrafuzzaman
Lead Software Engineer
NewsCred

(M) 880-175-5592433
Twitter | Blog | Facebook

Check out The Academy, your #1 source
for free content marketing resources

On Nov 26, 2014, at 6:23 PM, A.K.M. Ashrafuzzaman <ashrafuzzaman.g2@gmail.com> wrote:

> Hi guys,
> When we are using Kinesis with 1 shard then it works fine. But when we use more that
1 then it falls into an infinite loop and no data is processed by the spark streaming. In
the kinesis dynamo DB, I can see that it keeps increasing the leaseCounter. But it do start
processing.
> 
> I am using,
> scala: 2.10.4
> java version: 1.8.0_25
> Spark: 1.1.0
> spark-streaming-kinesis-asl: 1.1.0
> 
> A.K.M. Ashrafuzzaman
> Lead Software Engineer
> NewsCred
> 
> (M) 880-175-5592433
> Twitter | Blog | Facebook
> 
> Check out The Academy, your #1 source
> for free content marketing resources
> 


Mime
View raw message