kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MCG" <mcg-hel...@qq.com>
Subject 回复: KTable.suppress(Suppressed.untilWindowCloses) does not suppresssome non-final results when the kafka streams process is restarted
Date Wed, 26 Dec 2018 09:53:53 GMT
I'm not talking about orderliness, but that the same consumer group, the same partition, is
consumed by multiple consumers. I use kafka-consumer-groups.sh and org.apache.kafka.clients.admin.AdminClient
to validate the results. Because the same consumer group subscribes to a topic, the same partition
cannot be consumed by multiple consumers. But that happened.




------------------ 原始邮件 ------------------
发件人: "Peter Levart"<peter.levart@gmail.com>;
发送时间: 2018年12月26日(星期三) 下午5:21
收件人: "users"<users@kafka.apache.org>;"Guozhang Wang"<wangguoz@gmail.com>;

主题: Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppresssome non-final
results when the kafka streams process is restarted





On 12/21/18 3:16 PM, Peter Levart wrote:
> I also see some results that are actual non-final window aggregations 
> that precede the final aggregations. These non-final results are never 
> emitted out of order (for example, no such non-final result would ever 
> come after the final result for a particular key/window). 

Absence of proof is not the proof of absence... And I have later 
observed (using the DSL variant, not the custom Transformer) an 
occurrence of a non-final result that was emited after restart of 
streams processor while the final result for the same key/window had 
been emitted before the restart:

[pool-1-thread-4] APP Consumed: [a@1545815260000/1545815262000] -> [550, 
81, 18, 393, 968, 847, 452, 0, 0, 0], sum: 444856
...
... restart ...
...
[pool-1-thread-4] APP Consumed: [a@1545815260000/1545815262000] -> [550] 
INSTEAD OF [550, 81, 18, 393, 968, 847, 452, 0, 0, 0], sum: 551648


The app logic can not even rely on guarantee that results are ordered 
then. This is really not usable until the bug is fixed.

Regards, Peter
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message