samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: Monitoring consumer lag
Date Mon, 16 Nov 2015 18:02:26 GMT
The following metrics are reported via JMX.

1. Samza exposes a per-partition metric called "$topicname-$partition_name-
messages-behind-high-watermark". This measures the number of messages
behind the high watermark of your consumer. Ideally, at steady state, you
would expect this metric to be zero (with occasional spikes depending on
your traffic).

2. For your second usecase,  "$topicname-$partition_name-bytes-read",
""$topicname-$partition_name-messages-read" are interesting.

You can query these metrics and get alerted based on them.







On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr524@gmail.com>
wrote:

> Hi,
>
> I'd like to monitor consumer's lag.
> Found this tool https://github.com/linkedin/Burrow.
> But now realized that Samza is using it's own checkpointing mechanism.
>
> So question is what's the best way to monitor whether and how much the
> consumer is lagging?
>
> On a related subject, I'd also like to monitor throughput per topic in
> terms of messages per second and bytes per second. Should I query brokers
> periodically, or maybe there is a better way?
>
> Thanks,
> Michael
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message