samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Shkolnik <bor...@gmail.com>
Subject Re: Monitoring consumer lag
Date Tue, 17 Nov 2015 01:17:21 GMT
Just to add to Jagasish's suggestion - you can configure
MetricsSnapshotRecorder to output the metrics into a stream and read them
from there if it works better for you.

Boris.

On Mon, Nov 16, 2015 at 12:32 PM, Jordan Shaw <jordan@pubnub.com> wrote:

> Michael,
> I should have added if your using Burrow in the context of samza consumers
> it probably won't work because samza does it's own offset tracking (see
> checkpoint topics). The messages-behind-high-watermark is probably your
> best bet if you just want something out of the box and don't care about lag
> time only message count behind.
> -Jordan
>
> On Mon, Nov 16, 2015 at 11:40 AM, Michael Ravits <michaelr524@gmail.com>
> wrote:
>
> > Thanks Jagadish! I'll look further into this.
> >
> > Jordan, I tested Burrow with 0.8.3-SNAPSHOT and set it to read consumer
> > offsets from zookeeper because I assumed that it's the default Kafka
> config
> > for commiting offsets. Will try again with Burrow set to read from
> > __consumer_offsets.
> >
> > Thanks
> >
> > On Mon, Nov 16, 2015 at 8:04 PM, Jordan Shaw <jordan@pubnub.com> wrote:
> >
> > > Michael,
> > > It depends on how you define lag.
> > >
> > > 1) If you define lag as the total number of messages behind then burrow
> > is
> > > a good tool as long as all your infrastructure is on 0.8.2, it
> basically
> > > works by inspecting the __consumer_offsets topic which was introduced
> in
> > > 0.8.2 (they said they were going to support <0.8.2 but i don't think
> > that's
> > > a thing yet).
> > >
> > > 2. If you define lag in time units I would recommend sending a
> timestamp
> > > with the msg and doing some "manual" inspecting on the consumer end.
> > > -Jordan
> > >
> > > On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr524@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to monitor consumer's lag.
> > > > Found this tool https://github.com/linkedin/Burrow.
> > > > But now realized that Samza is using it's own checkpointing
> mechanism.
> > > >
> > > > So question is what's the best way to monitor whether and how much
> the
> > > > consumer is lagging?
> > > >
> > > > On a related subject, I'd also like to monitor throughput per topic
> in
> > > > terms of messages per second and bytes per second. Should I query
> > brokers
> > > > periodically, or maybe there is a better way?
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > >
> > >
> > >
> > > --
> > > Jordan Shaw
> > > Full Stack Software Engineer
> > > PubNub Inc
> > > 1045 17th St
> > > San Francisco, CA 94107
> > >
> >
>
>
>
> --
> Jordan Shaw
> Full Stack Software Engineer
> PubNub Inc
> 1045 17th St
> San Francisco, CA 94107
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message