kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parthasarathy, Mohan" <mpart...@hpe.com>
Subject Re: Can kafka internal state be purged ?
Date Tue, 18 Jun 2019 00:04:01 GMT
On further debugging, what we are seeing is that windows are expiring rather randomly as new
messages are being processed. . We tested with new key for every new message. We waited for
the window time before replaying new messages. Sometimes a new message would come in and create
state. It takes several messages to make some of the old windows to be closed (go past suppress
to the next stage). We have also seen where one of them never closed even but several other
older ones expired.  Then we explicitly sent a message with the same old key and then it showed
up. Also, for every new message, only one of the previous window expires even though there
are several pending.

If we don't use suppress, then there is never an issue. With suppress, the behavior we are
seeing is weird. We are using 2.1.0 version in DSL mode. Any clues on what we could be missing
? Why isn't there an order in the way windows are closed ? As event time progresses by the
new messages arriving, the older ones should expire. Is that right understanding or not ?

Thanks
Mohan

´╗┐On 6/17/19, 3:43 PM, "Parthasarathy, Mohan" <mparthas@hpe.com> wrote:

    Hi,
    
    We are using suppress in the application. We see some state being created at some point
in time. Now there is no new data for a day or two. We send new data but the old window of
data (where we see the state being created) is not closing i.e not seeing it go through suppress
and on to the next stage. It is as though the state created earlier was purged. Is this possible
?
    
    Thanks
    Mohan
    
    

Mime
View raw message