kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Powis <spo...@salesforce.com>
Subject Re: creating a controlled lag in for a consumer group
Date Mon, 04 Sep 2017 00:10:30 GMT
Hey Ido.

I haven't tried to do something like this, but we've worked out some future
plans to do something similar, so I have a bit of interest in what you're
saying.

I feel like there are some details left out from your post.  Are your
consumers able to keep up w/ the throughput without the sleep strategy in
place?  How are messages being published across partitions?  Evenly?  I
assume the timestamps on the messages are always increasing?  Where are the
timestamps generated? By the client publishers?  Are the clocks of all the
publishers sync'd?

Stephen

On Sun, Sep 3, 2017 at 8:01 PM, Ido Barkan <ido@appsflyer.com> wrote:

> Hey all.
> We are trying to create a controlled lag (a lag of 30 secs). We are doing
> that by inspecting a timestamp field for each msg on a specific topic and
> doing an actual Thread.sleep for a computed time in the consumer thread
> (until the msg is 30 secs old). We were hoping to see that eventually most
> of the messages are 30 seconds old and the created lag stands for 30 secs
> of processing rate.
> This works for us in low throughput but on high rates we are starting to
> witness strange behavior: lag on partitions is starting to grow in an
> uncontrollable manner and becomes uneven per partition.
> We have tried multiple variations of this: sleeping per bulk of msgs,
> sleeping per msg, capping the sleep time etc but failed to resolve this.
>
> Has anyone tried to do this? any ideas?
>
> Thanks,
> Ido
> --
> Thanks,
> Ido
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message