kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From François Langelier <f.langel...@gmail.com>
Subject Re: Disactivating Yammer Metrics Monitoring
Date Fri, 07 Nov 2014 15:24:51 GMT
Good to know!

Thanks Jason, I'll look at it ASAP! :)



François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Membre Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior

On Thu, Nov 6, 2014 at 12:30 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> Hi Francois,
>
> We had the exact same problem.  We embed Kafka in our service container,
> and we use yammer metrics to see data about the whole app (e.g. kafka, the
> jvm, the service container wrapping it).  However, as you observed, by
> default, kafka produces an insane amount of metrics.  So what we did, is
> using the yammer library, you can disable specific metrics by removing
> metrics from the yammer MetricsRegistry, which you can access from guice
> (if you are using guice).  I implemented the MetricsRegistryListener, and
> added the ability to remove metric names by regex, so I can still have some
> metrics show up (like the simple 'AllTopic' counts for messages/bytes sent
> from the producer), but block everything else that's per topic, etc....
>
> Jason
>
> On Fri, Sep 19, 2014 at 11:34 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > I don't have any source or configs handy to check things, but you are
> > saying you've configured Kafka to use GraphiteReporter, right?  So why
> not
> > remove that config, so metrics stop being sent to Graphite if your
> Graphite
> > setup is suffering?  If you do that and you still want to see your Kafka
> > metrics, you can always use SPM <http://sematext.com/spm/> for Kafka
> > (though some of the graphs will be empty until we get KAFKA-1481, or
> > something else that improves metrics, in).  If you just want to use it
> > temporarily, just use the free 30-day trial version until you beef up
> your
> > Graphite setup.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Fri, Sep 19, 2014 at 10:08 AM, François Langelier <
> > f.langelier@gmail.com>
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > Thank you for your answer.
> > >
> > > It's possible that I didn't understood something, if so correct me
> > please.
> > >
> > > From what I understood, from the kafka doc #monitoring
> > > <http://kafka.apache.org/documentation.html#monitoring>, kafka use
> > Yammer
> > > Metrics for monitoring the servers (the brokers) and the clients
> > (producers
> > > and consumers).
> > >
> > > Our web site is also using Yammer Metrics and push that to our Graphite
> > > server and our web site also produce message in kafka.
> > > From what I read, the Yammer Metrics GraphiteReporter is a kind of
> > > Singleton, once I Enable it, it is working for all the process. (But I
> > > might be wrong here...)
> > >
> > > We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade,
> > > kafka is monitoring in our Graphite Server and is hammering it, so we
> > > aren't able to use it because we always get timeout...
> > >
> > > SO, I was wondering if there is a way to disable the kafka monitoring
> to
> > > our Graphite server.
> > >
> > > We are using the code in the tag 0.8.1.1 on github, so if the
> > kafka-ganglia
> > > isn't in the tag, we aren't using it :)
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > > On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <
> desk@danielcompton.net
> > >
> > > wrote:
> > >
> > > > Hi Francois
> > > >
> > > > I didn't quite understand how you've set up your metrics reporting.
> Are
> > > you
> > > > using the https://github.com/criteo/kafka-ganglia metrics reporter?
> If
> > > so
> > > > then you should be able to adjust the config to exclude the metrics
> you
> > > > don't want, with kafka.ganglia.metrics.exclude.regex.
> > > >
> > > >
> > > > On 18 September 2014 07:55, François Langelier <
> f.langelier@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > We are using yammer metrics to monitor some parts of our system.
> > > > >
> > > > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more
> data
> > > > > getting in our graphite server and from what I saw, it looks like
> it
> > > all
> > > > > come from our producers.
> > > > >
> > > > > From what i understand, since we already use graphite, our
> > > > graphiteReporter
> > > > > is enable in our main web site and our kafka producers are having
> fun
> > > > using
> > > > > it too to monitor in graphite.
> > > > >
> > > > > The problem is that right now kafka is hammering of graphite server
> > and
> > > > we
> > > > > have difficulty to saw our monitored data...
> > > > >
> > > > > Is there a way to deactivate the monitoring of our kafka producers?
> > > > >
> > > > >
> > > > > François Langelier
> > > > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > > > <http://www.etsmtl.ca/>
> > > > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > > > VP-Communication - CS Games <http://csgames.org> 2014
> > > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > > > Comité Organisateur Olympiades ÉTS 2012
> > > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message