kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Hoover <roger.hoo...@gmail.com>
Subject Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service
Date Fri, 12 Jun 2015 18:34:02 GMT
Will do.  Thanks

Sent from my iPhone

> On Jun 12, 2015, at 10:43 AM, Todd Palino <tpalino@gmail.com> wrote:
> 
> Can you open an issue on the github page please, and we can investigate
> further there?
> 
> -Todd
> 
> On Fri, Jun 12, 2015 at 10:22 AM, Roger Hoover <roger.hoover@gmail.com>
> wrote:
> 
>> Thanks, Todd.  I'm also using ZK 3.4.6 with no ACLs.  I'm surprised and
>> currently stumped by this error.
>> 
>>> On Fri, Jun 12, 2015 at 9:49 AM, Todd Palino <tpalino@gmail.com> wrote:
>>> 
>>> The invalid ACL error is an error that is passed back from Zookeeper.
>> What
>>> version of Zookeeper are you using, and have you set up ACLs within it?
>> I'm
>>> not able to see this on our ZK (3.4.6 with no ACLs).
>>> 
>>> -Todd
>>> 
>>> On Fri, Jun 12, 2015 at 9:34 AM, Roger Hoover <roger.hoover@gmail.com>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I was trying to give burrow a try and got a ZK error "invalid ACL
>>>> specified".  Any suggestions on what's going wrong?
>>>> 
>>>> 1434044348908673512 [Critical] Cannot get ZK notifier lock: zk: invalid
>>> ACL
>>>> specified
>>>> 
>>>> 
>>>> Here's my config:
>>>> 
>>>> 
>>>> [general]
>>>> 
>>>> logdir=log
>>>> 
>>>> logconfig=logging.cfg
>>>> 
>>>> pidfile=burrow.pid
>>>> 
>>>> client-id=burrow-lagchecker
>>>> 
>>>> group-blacklist=^(console-consumer-|python-kafka-consumer-).*$
>>>> 
>>>> 
>>>> [zookeeper]
>>>> 
>>>> hostname=host1
>>>> 
>>>> port=2181
>>>> 
>>>> timeout=6
>>>> 
>>>> lock-path=/burrow/notifier
>>>> 
>>>> 
>>>> [kafka "sit"]
>>>> 
>>>> broker=host1
>>>> 
>>>> broker=host2
>>>> 
>>>> broker=host3
>>>> 
>>>> broker=host4
>>>> 
>>>> broker-port=9092
>>>> 
>>>> zookeeper=host1
>>>> 
>>>> zookeeper=host2
>>>> 
>>>> zookeeper=host3
>>>> 
>>>> zookeeper-port=2181
>>>> 
>>>> zookeeper-path=/
>>>> 
>>>> offsets-topic=__consumer_offsets
>>>> 
>>>> 
>>>> [tickers]
>>>> 
>>>> broker-offsets=60
>>>> 
>>>> 
>>>> [lagcheck]
>>>> 
>>>> intervals=10
>>>> 
>>>> expire-group=604800
>>>> 
>>>> 
>>>> [httpserver]
>>>> 
>>>> server=on
>>>> 
>>>> port=7000
>>>> 
>>>> On Tue, Jun 9, 2015 at 12:34 PM, Todd Palino <tpalino@gmail.com>
>> wrote:
>>>> 
>>>>> For mirror maker and our audit application, we've been using
>>>>> Kafka-committed offsets for some time now. We've got a few other
>>>> consumers
>>>>> who are using it, but we haven't actively worked on moving the bulk
>> of
>>>> them
>>>>> over. It's been less critical since we put the ZK transaction logs on
>>>> SSD.
>>>>> 
>>>>> And yeah, this is specific for kafka-committed offsets. I'm looking
>> at
>>>> some
>>>>> options for handling Zookeeper as well, but since our goal with this
>>> was
>>>> to
>>>>> monitor our own infrastructure applications and move forwards, it
>>> hasn't
>>>>> gotten a lot of my attention yet.
>>>>> 
>>>>> -Todd
>>>>> 
>>>>> 
>>>>>> On Tue, Jun 9, 2015 at 11:53 AM, Jason Rosenberg <jbr@squareup.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi Todd,
>>>>>> 
>>>>>> Thanks for open sourcing this, I'm excited to take a look.
>>>>>> 
>>>>>> It looks like it's specific to offsets stored in kafka (and not
>>>>> zookeeper)
>>>>>> correct?  I assume by that that LinkedIn is using the kafka storage
>>> now
>>>>> in
>>>>>> production?
>>>>>> 
>>>>>> Jason
>>>>>> 
>>>>>> On Thu, Jun 4, 2015 at 9:43 PM, Todd Palino <tpalino@gmail.com>
>>> wrote:
>>>>>> 
>>>>>>> I am very happy to introduce Burrow, an application to provide
>>> Kafka
>>>>>>> consumer status as a service. Burrow is different than just a
>> "lag
>>>>>>> checker":
>>>>>>> 
>>>>>>> * Multiple Kafka cluster support - Burrow supports any number
of
>>>> Kafka
>>>>>>> clusters in a single instance. You can also run multiple copies
>> of
>>>>> Burrow
>>>>>>> in parallel and only one of them will send out notifications.
>>>>>>> 
>>>>>>> * All consumers, all partitions - If the consumer is committing
>>>> offsets
>>>>>> to
>>>>>>> Kafka (not Zookeeper), it will be available in Burrow
>>> automatically.
>>>>>> Every
>>>>>>> partition it consumes will be monitored simultaneously, avoiding
>>> the
>>>>> trap
>>>>>>> of just watching the worst partition (MaxLag) or spot checking
>>>>> individual
>>>>>>> topics.
>>>>>>> 
>>>>>>> * Status can be checked via HTTP request - There's an internal
>> HTTP
>>>>>> server
>>>>>>> that provides topic and consumer lists, can give you the latest
>>>> offsets
>>>>>> for
>>>>>>> a topic either from the brokers or from the consumer, and lets
>> you
>>>>> check
>>>>>>> consumer status.
>>>>>>> 
>>>>>>> * Continuously monitor groups with output via email or a call
to
>> an
>>>>>>> external HTTP endpoint - Configure emails to send for bad groups,
>>>>> checked
>>>>>>> continuously. Or you can have Burrow call an HTTP endpoint into
>>>> another
>>>>>>> system for handling alerts.
>>>>>>> 
>>>>>>> * No thresholds - Status is determined over a sliding window
and
>>> does
>>>>> not
>>>>>>> rely on a fixed limit. When a consumer is checked, it has a
>> status
>>>>>>> indicator that tells whether it is OK, a warning, or an error,
>> and
>>>> the
>>>>>>> partitions that caused it to be bad are provided.
>>>>>>> 
>>>>>>> 
>>>>>>> Burrow was created to address specific problems that LinkedIn
has
>>>> with
>>>>>>> monitoring consumers, in particular wildcard consumers like
>> mirror
>>>>> makers
>>>>>>> and our audit consumers. Instead of checking offsets for specific
>>>>>> consumers
>>>>>>> periodically, it monitors the stream of all committed offsets
>>>>>>> (__consumer_offsets) and continually calculates lag over a
>> sliding
>>>>>> window.
>>>>>>> 
>>>>>>> We welcome all feedback, comments, and contributors. This project
>>> is
>>>>> very
>>>>>>> much under active development for us (we're using it in some
of
>> our
>>>>>>> environments now, and working on getting it running everywhere
to
>>>>> replace
>>>>>>> our previous monitoring system).
>>>>>>> 
>>>>>>> Burrow is written in Go, published under the Apache License,
and
>>>> hosted
>>>>>> on
>>>>>>> GitHub at:
>>>>>>> https://github.com/linkedin/Burrow
>>>>>>> 
>>>>>>> Documentation is on the GitHub wiki at:
>>>>>>> https://github.com/linkedin/Burrow/wiki
>>>>>>> 
>>>>>>> -Todd
>> 

Mime
View raw message