spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <shushantaror...@gmail.com>
Subject receiver based spark streaming doubts
Date Sun, 13 Nov 2016 19:04:54 GMT
Hi

In spark streaming based on receivers - when receiver gets data and store
in blocks for workers to process, How many blocks does receiver gives to
worker.

Say I have a streaming app with 30 sec of batch interval what will happen
1.for first batch(first 30 sec) there will not be any data for worker to
process, only receiver will fetch data and store in blocks
2.for second batch- worker will work on block fetched in step1 and
receibvers will fetch more data

Is this understanding correct ? In that case I have worst case of 2*batch
interval delay in event processing .

2.Also what if worker is slow - say in above example of 30 sec batch
interval - worker took 2 min to process a batch then in next batch will it
get 4 blocks(2 min data fetch by receiver) or just the 1 batch interval
data irrespective of its speed.

Thanks

Mime
View raw message