samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Ravits <michaelr...@gmail.com>
Subject Re: Monitoring consumer lag
Date Mon, 16 Nov 2015 19:40:31 GMT
Thanks Jagadish! I'll look further into this.

Jordan, I tested Burrow with 0.8.3-SNAPSHOT and set it to read consumer
offsets from zookeeper because I assumed that it's the default Kafka config
for commiting offsets. Will try again with Burrow set to read from
__consumer_offsets.

Thanks

On Mon, Nov 16, 2015 at 8:04 PM, Jordan Shaw <jordan@pubnub.com> wrote:

> Michael,
> It depends on how you define lag.
>
> 1) If you define lag as the total number of messages behind then burrow is
> a good tool as long as all your infrastructure is on 0.8.2, it basically
> works by inspecting the __consumer_offsets topic which was introduced in
> 0.8.2 (they said they were going to support <0.8.2 but i don't think that's
> a thing yet).
>
> 2. If you define lag in time units I would recommend sending a timestamp
> with the msg and doing some "manual" inspecting on the consumer end.
> -Jordan
>
> On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr524@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'd like to monitor consumer's lag.
> > Found this tool https://github.com/linkedin/Burrow.
> > But now realized that Samza is using it's own checkpointing mechanism.
> >
> > So question is what's the best way to monitor whether and how much the
> > consumer is lagging?
> >
> > On a related subject, I'd also like to monitor throughput per topic in
> > terms of messages per second and bytes per second. Should I query brokers
> > periodically, or maybe there is a better way?
> >
> > Thanks,
> > Michael
> >
>
>
>
> --
> Jordan Shaw
> Full Stack Software Engineer
> PubNub Inc
> 1045 17th St
> San Francisco, CA 94107
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message