spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Re: [SS] FlatMapGroupsWithStateExec with no commitTimeMs metric?
Date Mon, 26 Nov 2018 07:41:02 GMT
Just filed a new JIRA issue [1] as well as PR [2].

- Jungtaek Lim (HeartSaVioR)

1. https://issues.apache.org/jira/browse/SPARK-26170
2. https://github.com/apache/spark/pull/23142


2018년 11월 26일 (월) 오후 3:26, Jungtaek Lim <kabhwan@gmail.com>님이 작성:

> Looks like just a kind of missing spot. Just crafted the patch and now in
> progress of testing.
>
> Once done with testing I'll file a new issue and submit a patch.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 11월 26일 (월) 오전 4:49, Burak Yavuz <brkyvz@gmail.com>님이 작성:
>
>> Probably just oversight. Anyone is welcome to add it :)
>>
>> On Sun, Nov 25, 2018 at 8:55 AM Jacek Laskowski <jacek@japila.pl> wrote:
>>
>>> Hi,
>>>
>>> Why is FlatMapGroupsWithStateExec not measuring the time taken on state
>>> commit [1](like StreamingDeduplicateExec [2] and StreamingGlobalLimitExec
>>> [3])? Is this on purpose? Why? Thanks.
>>>
>>> [1]
>>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala#L135
>>>
>>> [2]
>>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L482
>>>
>>> [3]
>>> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingGlobalLimitExec.scala#L87
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://about.me/JacekLaskowski
>>> Mastering Spark SQL https://bit.ly/mastering-spark-sql
>>> Spark Structured Streaming https://bit.ly/spark-structured-streaming
>>> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>

Mime
View raw message