kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Barr <dave.b...@gmail.com>
Subject Re: Use Kafka to collect JVM and JMX MBean's statistics.
Date Mon, 09 Apr 2012 16:43:01 GMT
We have an agent (tcollector) that collects various metrics and sends
them to OpenTSDB.  We then have a simple python script that lets you
query OpenTSDB for a given metric and alert on a threshold.  We
currently gather about 1300 metrics and about 10000 datapoints/sec.

https://github.com/stumbleupon/tcollector
http://opentsdb.net/nagios.html

We're just starting to use Kafka here.  I'm not aware yet of a Kafka
collector, but I'm sure one will be coming soon.  They're really
simple to write.  You can look at either the elasticsearch collector
or the HBase collector as an example of different styles.  The HBase
one forks a jmx process to read metrics from the running JVM, the
elasticsearch one queries the running process for metrics via an
ElasticSearch endpoint.

--Dave
On Mon, Apr 9, 2012 at 12:30 AM, liu brent <liubaochun@gmail.com> wrote:
> Hi,
>  Currently, there are 20+ kinds of web service running on hundreds of
> machines in 4 data centers, which are written in java. We want to collect
> and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> and machines, and then send the waring to Nagios.  Does anyone has the
> experience in using Kafka to do so?
>  The direct solution seems like that we install a client on every machine,
> which works as JMX client, collect the information and work as kafka
> producer. Anyone has better ideas or suggestions.
>
> Thanks,
> Liu

Mime
View raw message