kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parthasarathy, Mohan" <mpart...@hpe.com>
Subject Re: Can kafka internal state be purged ?
Date Wed, 19 Jun 2019 19:49:40 GMT
Thanks. We will give it a shot.

´╗┐On 6/19/19, 12:42 PM, "Bruno Cadonna" <bruno@confluent.io> wrote:

    Hi Mohan,
    
    I realized that my previous statement was not clear. With a grace
    period of 12 hour, suppress would wait for late events until stream
    time has advanced 12 hours before a result would be emitted.
    
    Best,
    Bruno
    
    On Wed, Jun 19, 2019 at 9:21 PM Bruno Cadonna <bruno@confluent.io> wrote:
    >
    > Hi Mohan,
    >
    > if you do not set a grace period, the grace period defaults to 12
    > hours. Hence, suppress would wait for an event that occurs 12 hour
    > later before it outputs a result. Try to explicitly set the grace
    > period to 0 and let us know if it worked.
    >
    > If it still does not work, upgrade to version 2.2.1 if it is possible
    > for you. We had a couple of bugs in suppress recently that are fixed
    > in that version.
    >
    > Best,
    > Bruno
    >
    > On Wed, Jun 19, 2019 at 8:37 PM Parthasarathy, Mohan <mparthas@hpe.com> wrote:
    > >
    > > No, I have not set any grace period. Is that mandatory ? Have you seen problems
with suppress and windows expiring ?
    > >
    > > Thanks
    > > Mohan
    > >
    > > On 6/19/19, 12:41 AM, "Bruno Cadonna" <bruno@confluent.io> wrote:
    > >
    > >     Hi Mohan,
    > >
    > >     Did you set a grace period on the window?
    > >
    > >     Best,
    > >     Bruno
    > >
    > >     On Tue, Jun 18, 2019 at 2:04 AM Parthasarathy, Mohan <mparthas@hpe.com>
wrote:
    > >     >
    > >     > On further debugging, what we are seeing is that windows are expiring
rather randomly as new messages are being processed. . We tested with new key for every new
message. We waited for the window time before replaying new messages. Sometimes a new message
would come in and create state. It takes several messages to make some of the old windows
to be closed (go past suppress to the next stage). We have also seen where one of them never
closed even but several other older ones expired.  Then we explicitly sent a message with
the same old key and then it showed up. Also, for every new message, only one of the previous
window expires even though there are several pending.
    > >     >
    > >     > If we don't use suppress, then there is never an issue. With suppress,
the behavior we are seeing is weird. We are using 2.1.0 version in DSL mode. Any clues on
what we could be missing ? Why isn't there an order in the way windows are closed ? As event
time progresses by the new messages arriving, the older ones should expire. Is that right
understanding or not ?
    > >     >
    > >     > Thanks
    > >     > Mohan
    > >     >
    > >     > On 6/17/19, 3:43 PM, "Parthasarathy, Mohan" <mparthas@hpe.com>
wrote:
    > >     >
    > >     >     Hi,
    > >     >
    > >     >     We are using suppress in the application. We see some state being
created at some point in time. Now there is no new data for a day or two. We send new data
but the old window of data (where we see the state being created) is not closing i.e not seeing
it go through suppress and on to the next stage. It is as though the state created earlier
was purged. Is this possible ?
    > >     >
    > >     >     Thanks
    > >     >     Mohan
    > >     >
    > >     >
    > >     >
    > >
    > >
    

Mime
View raw message