kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parthasarathy, Mohan" <mpart...@hpe.com>
Subject Re: Can kafka internal state be purged ?
Date Wed, 19 Jun 2019 18:37:22 GMT
No, I have not set any grace period. Is that mandatory ? Have you seen problems with suppress
and windows expiring ?


´╗┐On 6/19/19, 12:41 AM, "Bruno Cadonna" <bruno@confluent.io> wrote:

    Hi Mohan,
    Did you set a grace period on the window?
    On Tue, Jun 18, 2019 at 2:04 AM Parthasarathy, Mohan <mparthas@hpe.com> wrote:
    > On further debugging, what we are seeing is that windows are expiring rather randomly
as new messages are being processed. . We tested with new key for every new message. We waited
for the window time before replaying new messages. Sometimes a new message would come in and
create state. It takes several messages to make some of the old windows to be closed (go past
suppress to the next stage). We have also seen where one of them never closed even but several
other older ones expired.  Then we explicitly sent a message with the same old key and then
it showed up. Also, for every new message, only one of the previous window expires even though
there are several pending.
    > If we don't use suppress, then there is never an issue. With suppress, the behavior
we are seeing is weird. We are using 2.1.0 version in DSL mode. Any clues on what we could
be missing ? Why isn't there an order in the way windows are closed ? As event time progresses
by the new messages arriving, the older ones should expire. Is that right understanding or
not ?
    > Thanks
    > Mohan
    > On 6/17/19, 3:43 PM, "Parthasarathy, Mohan" <mparthas@hpe.com> wrote:
    >     Hi,
    >     We are using suppress in the application. We see some state being created at
some point in time. Now there is no new data for a day or two. We send new data but the old
window of data (where we see the state being created) is not closing i.e not seeing it go
through suppress and on to the next stage. It is as though the state created earlier was purged.
Is this possible ?
    >     Thanks
    >     Mohan

View raw message