kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ido Barkan <...@appsflyer.com>
Subject creating a controlled lag in for a consumer group
Date Sun, 03 Sep 2017 11:01:41 GMT
Hey all.
We are trying to create a controlled lag (a lag of 30 secs). We are doing
that by inspecting a timestamp field for each msg on a specific topic and
doing an actual Thread.sleep for a computed time in the consumer thread
(until the msg is 30 secs old). We were hoping to see that eventually most
of the messages are 30 seconds old and the created lag stands for 30 secs
of processing rate.
This works for us in low throughput but on high rates we are starting to
witness strange behavior: lag on partitions is starting to grow in an
uncontrollable manner and becomes uneven per partition.
We have tried multiple variations of this: sleeping per bulk of msgs,
sleeping per msg, capping the sleep time etc but failed to resolve this.

Has anyone tried to do this? any ideas?

Thanks,
Ido
-- 
Thanks,
Ido

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message