spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SS] Bug in StreamExecution? currentBatchId and getBatchDescriptionString for web UI
Date Sat, 09 Sep 2017 19:21:33 GMT

While reviewing StreamExecution and how batches are displayed in web
UI, I've noticed that currentBatchId is -1 when StreamExecution is
created [1] and becomes 0 when no offsets are available [2].

That leads to my question about setting the job description for a
query using getBatchDescriptionString [3]. It branches per
currentBatchId and when it's -1 gives "init" [4] which never happens
as showed above.

That leads to the PR for SPARK-20464 "Add a job group and description
for streaming queries and fix cancellation of running jobs using the
job group" that sets the job description after populateStartOffsets

Shouldn't it be before populateStartOffsets so
getBatchDescriptionString has a chance of giving "init" and we see no
two 0s?

Help appreciated.


Jacek Laskowski
Spark Structured Streaming (Apache Spark 2.2+)
Mastering Apache Spark 2
Follow me at

To unsubscribe e-mail:

View raw message