kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwilym Evans <gwilym.ev...@bigcommerce.com>
Subject Reasonable time to commit offsets?
Date Wed, 14 Dec 2016 10:17:24 GMT
Hi folks,

New to the list and new to operating Kafka. I'm trying to find out what a
reasonable turnaround time for committing offsets is.

I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper
nodes, though the cluster has been upgraded from its starting point at
0.10.0.0. The offsets topic is at its default configuration. The partition
I'm consuming and committing offsets for has 150 partitions.

While I can steadily consume around 90,000 messages/sec using 1 consumer
without (synchronously) committing offsets, if I want to do so
synchronously it takes between 1000ms and 2000ms to complete the commit. If
I use multiple consumers to reduce the partition count each is assigned
(I've tested up to 10) then time-to-commit does _not_ change.

Is this reasonable? Or should it be quicker than this?

For now, my use case is simple topic-splitting: read messages, apply basic
filters, discard or resend messages to one or more topics. But I don't want
to advance the offsets until I am certain the messages have been written
back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit.

If more information is needed, I'm happy to supply what I can.

Thanks,
Gwilym

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message