kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: Ganglia Metrics Reporter
Date Fri, 01 Nov 2013 18:07:36 GMT

It looks as though adding multicast support is more an issue with the Metrics code rather
than with kafka-ganglia.  The only way I could see to tell GangliaReporter to use multicast
when sending messages was to pass it a GangliaMessageBuilder that constructed GangliaMessages
using a proper MulticastSocket which sets the multicast TTL.

I attempted to do this like this:

However, GangliaMessageBuilder is not a public class, so I can't extend it as MulticastGangliaMessageBuilder.

I'm not sure what to do at this point.  I'd rather use this library but, it looks like I might
have to go with jmxtrans after all.  Thoughts?


On Oct 29, 2013, at 2:38 PM, Maxime Brugidou <maxime.brugidou@gmail.com> wrote:

> Hi Andrew, how do you plan to use metrics 3.0.1? The current Kafka 0.8
> version uses 2.2 AFAIK so this is going to require a new Kafka version.
> I don't really use multicast TTL muself but you are right that this should
> be configurable and I'll definitely accept pull request going in that
> direction. It's a tiny piece of code so it shouldn't be hard to dig it.
> Thanks for using this small plugin and let me know what you plan to do.
> Cheers
> On Oct 29, 2013 2:00 PM, "Andrew Otto" <otto@wikimedia.org> wrote:
>> Hi Maxime,
>> I'm using this at the Wikimedia Foundation to send Kafka Broker metrics to
>> Ganglia.  However, we use Ganglia in multicast mode.  This mostly seems to
>> work with your code, but the ttl on the multicast packets gets set at 1.
>> We have sometimes have multiple levels of ganglia aggregators, and with
>> ttl=1, the multicast packets don't make it to the proper aggregators.
>> I'm looking into either forking or rewriting this library using Codahale
>> Metrics v 3.0.1, and supporting multicast more explicitly.  Is this
>> something you could do better/faster than me, or should I proceed? :)
>> -Andrew Otto
>> (Thanks for writing this, btw!)
>> On Aug 22, 2013, at 11:42 AM, Maxime Brugidou <maxime.brugidou@gmail.com>
>> wrote:
>>> Hi all,
>>> Since I couldn't find any other way to publish kafka metrics to ganglia
>>> from kafka 0.8 (beta), I just published on github a super-simple ganglia
>>> metrics reporter for Kafka. It is configurable through the kafka config
>>> file and you can use it on the broker side and on your
>> consumers/producers.
>>> There is also a feature to exclude some metrics with a regex (useful if
>> you
>>> have many topics/partions).
>>> Here it is on github: https://github.com/criteo/kafka-ganglia
>>> Let me know if you have issues/questions. This is using metrics-ganglia
>>> 2.2.0 directly so it is not a ganglia plugin but rather a kafka "add-on".
>>> I don't know the proper way to distribute this yet for installation on
>>> brokers, it could also be part of some contrib kafka code.
>>> Cheers,
>>> Maxime

View raw message