spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-29312) Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators
Date Tue, 01 Oct 2019 09:03:00 GMT
Jungtaek Lim created SPARK-29312:
------------------------------------

             Summary: Inconsistent behavior on metrics when dropping rows in state between
FlatMapGroupWithState and other stateful operators
                 Key: SPARK-29312
                 URL: https://issues.apache.org/jira/browse/SPARK-29312
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


Even rows are evicted from state, we don't increase the value of "updated" metrics. The rule
applies on streaming aggregations as well as streaming join, but FlatMapGroupsWithState increases
the value of "updated" metrics, allowing the metrics to be shown like (total = 0, updated
= 1) which looks weird for who are familiar with metrics with others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message