spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <>
Subject spark streaming 1.3 with kafka
Date Tue, 01 Sep 2015 13:02:41 GMT

In spark streaming 1.3 with kafka- when does driver bring latest offsets of
this run - at start of each batch or at time when  batch gets queued ?

Say few of my batches take longer time to complete than their batch
interval. So some of batches will go in queue. Will driver waits for
 queued batches to get started or just brings the latest offsets before
they even actually started. And when they start running they will work on
old offsets brought at time when they were queued.

View raw message