spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SS] number of output rows metric for streaming aggregation (StateStoreSaveExec) in Append output mode not measured?
Date Sat, 12 Oct 2019 16:12:18 GMT

I use Spark 2.4.4

I've just noticed that the number of output rows metric
of StateStoreSaveExec physical operator does not seem to be measured for
Append output mode. In other words, whatever happens before or
after StateStoreSaveExec operator the metric is always 0.

It is measured for the other modes - Complete and Update.


Is this intentional? Why?

Jacek Laskowski
The Internals of Spark SQL
The Internals of Spark Structured Streaming
The Internals of Apache Kafka
Follow me at

View raw message