spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Aggarwal <>
Subject submissionTime vs batchTime, DirectKafka
Date Wed, 09 Mar 2016 17:09:28 GMT
Hi All,

we have batchTime and submissionTime.

@param batchTime   Time of the batch

@param submissionTime  Clock time of when jobs of this batch was
submitted to the streaming scheduler queue

1) we are seeing difference between batchTime and submissionTime for small
batches(300ms) even in minutes for direct kafka this we see, only when the
processing time is more than the batch interval. how can we explain this

2) In one of case batch processing time is more then batch interval, then
will spark fetch the next batch data from kafka parallelly processing the
current batch or it will wait for current batch to finish first ?

I would be thankful if you give me some pointers


Thanks & Regards

Sachin Aggarwal

View raw message