kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Palino <tpal...@gmail.com>
Subject [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service
Date Fri, 05 Jun 2015 01:43:12 GMT
I am very happy to introduce Burrow, an application to provide Kafka
consumer status as a service. Burrow is different than just a "lag checker":

* Multiple Kafka cluster support - Burrow supports any number of Kafka
clusters in a single instance. You can also run multiple copies of Burrow
in parallel and only one of them will send out notifications.

* All consumers, all partitions - If the consumer is committing offsets to
Kafka (not Zookeeper), it will be available in Burrow automatically. Every
partition it consumes will be monitored simultaneously, avoiding the trap
of just watching the worst partition (MaxLag) or spot checking individual
topics.

* Status can be checked via HTTP request - There's an internal HTTP server
that provides topic and consumer lists, can give you the latest offsets for
a topic either from the brokers or from the consumer, and lets you check
consumer status.

* Continuously monitor groups with output via email or a call to an
external HTTP endpoint - Configure emails to send for bad groups, checked
continuously. Or you can have Burrow call an HTTP endpoint into another
system for handling alerts.

* No thresholds - Status is determined over a sliding window and does not
rely on a fixed limit. When a consumer is checked, it has a status
indicator that tells whether it is OK, a warning, or an error, and the
partitions that caused it to be bad are provided.


Burrow was created to address specific problems that LinkedIn has with
monitoring consumers, in particular wildcard consumers like mirror makers
and our audit consumers. Instead of checking offsets for specific consumers
periodically, it monitors the stream of all committed offsets
(__consumer_offsets) and continually calculates lag over a sliding window.

We welcome all feedback, comments, and contributors. This project is very
much under active development for us (we're using it in some of our
environments now, and working on getting it running everywhere to replace
our previous monitoring system).

Burrow is written in Go, published under the Apache License, and hosted on
GitHub at:
https://github.com/linkedin/Burrow

Documentation is on the GitHub wiki at:
https://github.com/linkedin/Burrow/wiki

-Todd

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message