kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apurva Mehta <apu...@confluent.io>
Subject Re: Reasonable time to commit offsets?
Date Wed, 14 Dec 2016 18:55:20 GMT
Hi Gwilym,

What is the latency for synchronously producing to this cluster? Is it also
1000 to 2000ms?

Thanks,
Apurva

On Wed, Dec 14, 2016 at 2:17 AM, Gwilym Evans <gwilym.evans@bigcommerce.com>
wrote:

> Hi folks,
>
> New to the list and new to operating Kafka. I'm trying to find out what a
> reasonable turnaround time for committing offsets is.
>
> I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper
> nodes, though the cluster has been upgraded from its starting point at
> 0.10.0.0. The offsets topic is at its default configuration. The partition
> I'm consuming and committing offsets for has 150 partitions.
>
> While I can steadily consume around 90,000 messages/sec using 1 consumer
> without (synchronously) committing offsets, if I want to do so
> synchronously it takes between 1000ms and 2000ms to complete the commit. If
> I use multiple consumers to reduce the partition count each is assigned
> (I've tested up to 10) then time-to-commit does _not_ change.
>
> Is this reasonable? Or should it be quicker than this?
>
> For now, my use case is simple topic-splitting: read messages, apply basic
> filters, discard or resend messages to one or more topics. But I don't want
> to advance the offsets until I am certain the messages have been written
> back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit.
>
> If more information is needed, I'm happy to supply what I can.
>
> Thanks,
> Gwilym
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message