kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Palino <tpal...@gmail.com>
Subject Re: Delay in Zookeeper Offset updates
Date Thu, 17 Sep 2015 20:25:18 GMT
Consumer offsets in Zookeeper are not handled by the Kafka brokers at all -
the consumer writes those directly to Zookeeper. Most likely, what you are
seeing is the interval over which the consumer is committing offsets.
Assuming that you are using the auto.commit.enable setting (it defaults to
true, so unless you have changed that in the consumer configuration it will
apply here), you should check the setting of auto.commit.interval for your
consumer. I believe the default is still 60 seconds (or a config value of
60000). If you're seeing the values change in ZK every 5-6 seconds, then it
sounds like it is set much lower.

The setting for the offset commit interval is entirely up to you. The lower
the value, the more often offsets are checkpointed to Zookeeper. However
that means that your consumer is writing to Zookeeper more often, and that
has performance impacts. In an environment like ours, where there is a
large number of consumers for a given cluster, we actually had to increase
the interval to 10 minutes for quite some time because the load on
Zookeeper was just too high. Once we moved the ZK transaction logs to SSDs,
we were able to drop that back to 60 seconds.

-Todd



On Thu, Sep 17, 2015 at 1:14 PM, nitin sharma <kumarsharma.nitin@gmail.com>
wrote:

> anyone faced this issue?
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Wed, Sep 2, 2015 at 5:32 PM, nitin sharma <kumarsharma.nitin@gmail.com>
> wrote:
>
> > Hi All,
> >
> > i have run into a weird issue with my Kafka setup.
> >
> > I see that it takes around 5-6 sec for Zookeeper to update the offset for
> > Kafka topics.
> >
> > I am running "ConsumerOffsetChecker" tool to see that lag and what i
> found
> > that even when my consumer is not up, it takes 5-6 sec for Zookeeper to
> > show the updated Offset .
> >
> > Is this behavior can be fixed? I have tried adding following parameters
> in
> > my Kafka server.property files but no luck.
> >
> > log.flush.interval.messages=1
> > log.default.flush.scheduler.interval.ms=500
> >
> >
> > ConsumerOffsetChecker:
> >  bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect
> > localhost:2181 --group perfConsumer
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message