kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Yeargers <jon.yearg...@cedexis.com>
Subject Re: lag monitoring
Date Mon, 05 Dec 2016 15:07:57 GMT
FWIW - solved this by calling '.poll()'  with 'enable.auto.commit' set to
false.

On Mon, Dec 5, 2016 at 5:53 AM, Mathieu Fenniak <
mathieu.fenniak@replicon.com> wrote:

> Hi Jon,
>
> Here are some lag monitoring options that are external to the consumer
> application itself; I don't know if these will be appropriate for you.  You
> can use a command-line tool like kafka-consumer-groups.sh to monitor
> consumer group lag externally (
> http://kafka.apache.org/documentation.html#basic_ops_consumer_group), you
> can use Kafka JMX metrics to monitor lag (
> http://kafka.apache.org/documentation.html#monitoring), or you can use an
> external tool like Burrow (https://github.com/linkedin/Burrow).
>
> Mathieu
>
>
>
> On Mon, Dec 5, 2016 at 4:47 AM, Jon Yeargers <jon.yeargers@cedexis.com>
> wrote:
>
> > Is there a way to get updated consumer position(s) without subscribing
> to a
> > topic? I can achieve this by continually closing / reopening a
> > KafkaConsumer object but this is problematic as it often times out.
> >
> > Im getting consumer lag from a combination of
> >
> > (start) .seekToEnd() (and then) .position()
> >
> > and
> >
> > (end) .committed()
> >
> > but if I wait and call this set of functions again I get the same values.
> >
> > How do I refresh this connection without interfering with the actual
> > consumer group position(s)?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message