spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject [SS] number of output rows metric for streaming aggregation (StateStoreSaveExec) in Append output mode not measured?
Date Sat, 12 Oct 2019 16:12:18 GMT
Hi,

I use Spark 2.4.4

I've just noticed that the number of output rows metric
of StateStoreSaveExec physical operator does not seem to be measured for
Append output mode. In other words, whatever happens before or
after StateStoreSaveExec operator the metric is always 0.

It is measured for the other modes - Complete and Update.

See
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L329-L365

Is this intentional? Why?

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Mime
View raw message