kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apostolis Glenis <apostgle...@gmail.com>
Subject Re: Kafka Monitoring
Date Thu, 14 Dec 2017 14:30:01 GMT
I have also created a monitoring application for Kafka that uses prometheus.
You can look at the source code here:

https://github.com/aglenis/kafka_monitoring_pandas

2017-12-13 9:53 GMT+02:00 Irtiza Ali <iali@an10.io>:

> Ok thank you Michal
>
> On Tue, Dec 12, 2017 at 9:30 PM, Michal Michalski <
> michal.michalski@zalando.ie> wrote:
>
> > Hi Irtiza,
> >
> > Unfortunately I don't know what could be the issue in your case - in my
> > case all these metrics simply appear under appropriate keys
> > (e.g. "kafka.controller") in the JSON response of the /list request - the
> > same as the one you're making (I checked it right now to be sure). I also
> > checked our Kafka broker configuration and we don't seem to use any
> options
> > related to metrics reporting (I don't even know if there are any...).
> > Reading the Kafka documentation I'm under impression that all the metrics
> > should be available to you out-of-the-box, so I'm not sure why you can't
> > see them, sorry :-(
> >
> > M.
> >
> >
> > On 11 December 2017 at 14:11, Irtiza Ali <iali@an10.io> wrote:
> >
> > > Hi Michal,
> > >
> > > I have followed the steps below for the configuring Jolokia with kafka
> > >
> > >    -
> > >
> > >    Install the jolokia JVM-Agent jar from their site.
> > >    -
> > >
> > >    Copy the jolokia jar file in the  /opt/jolokia-agent/, create
> > > jolokia-agent
> > >    directory as it doesn’t exist already.
> > >    -
> > >
> > >    Move to the directory where your Kafka exists run the following
> > command
> > >
> > > $ export KAFKA_OPTS="$KAFKA_OPTS
> > > -javaagent:/opt/jolokia-agent/jolokia-jvm-1.3.7-agent.jar"
> > >
> > >    -
> > >
> > >    To expose the kafka metrics open the this file
> > >    /kafka/bin/kafka-server-start.sh file and find this (exec
> > >    $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@") line and
> > > paste
> > >    the line given below above it.
> > >
> > >
> > > export JMX_PORT=${JMX_PORT:-9999}
> > >
> > >    -
> > >
> > >    Restart the zookeeper and broker
> > >
> > >
> > > I think the issue is with the way I am exposing the metrics. I followed
> > the
> > > following link to enable JMX on kafka broker (
> > > https://stackoverflow.com/questions/36708384/enable-jmx-
> on-kafka-brokers
> > ).
> > > The issue that I am still facing is that when I list all the metrics
> > using
> > > (
> > > http://localhost:9999/jolokia/list) address. i dont get metrics
> related
> > > to:
> > > <host>:8074/jolokia/read/kafka.server:*
> > > <host>:8074/jolokia/read/kafka.controller:*
> > > <host>:8074/jolokia/read/kafka.log:*
> > > <host>:8074/jolokia/read/kafka.network:*
> > >
> > >
> > > Can you check what I have done wrong?
> > >
> > > Thanks
> > >
> > > With Regards
> > > Irtiza
> > >
> > > On Fri, Dec 8, 2017 at 8:19 PM, Michal Michalski <
> > > michal.michalski@zalando.ie> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have no modifications in that file - what we do is having a
> > "wrapper"
> > > > that's just a Docker "entrypoint" (just a bash script) which contents
> > is:
> > > >
> > > > export KAFKA_OPTS="$KAFKA_OPTS
> > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0"
> > > > exec ${KAFKA_DIR}/bin/kafka-server-start.sh
> > > > ${KAFKA_DIR}/config/server.properties
> > > >
> > > > Regarding your second question - we have an in-house monitoring
> > solution
> > > > that allows querying that endpoint and extracting metrics from the
> JSON
> > > > returned.
> > > >
> > > > If you're asking about what you should monitor, I think these links
> > will
> > > > answer your question:
> > > > https://docs.confluent.io/3.0.0/kafka/monitoring.html
> > > > https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
> > > >
> > > >
> > > > On 8 December 2017 at 14:56, Irtiza Ali <iali@an10.io> wrote:
> > > >
> > > > > Thanks Michal, can you kindly send me you kafka-run-class.sh and
> > > > > kafka-server-start.sh file, I have look what have you done.
> Because I
> > > > have
> > > > > done same thing that you explained above but when i do this <
> > > > > http://localhost:9999/jolokia/list> i get only metrics for the
> > > zookeeper
> > > > > but not the above metrics.
> > > > >
> > > > > How are you using Jolokia for monitoring kafka cluster??
> > > > >
> > > > > Thanks in advance
> > > > >
> > > > > On Fri, Dec 8, 2017 at 3:10 PM, Michal Michalski <
> > > > > michal.michalski@zalando.ie> wrote:
> > > > >
> > > > > > Hi Irtiza,
> > > > > >
> > > > > > I don't have any tutorial, but I can tell you what we do :-)
> > > > > >
> > > > > > First of all we have Jolokia agent jar included in our Kafka
> Docker
> > > > > image.
> > > > > > Then we append this to KAFKA_OPTS
> > > > > >
> > > > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0
> > > > > >
> > > > > >
> > > > > > Relevant port is then "exposed" in Docker image and "allowed"
in
> > AWS
> > > > > > Security Group.
> > > > > > Then our monitoring tool is querying the following endpoins
> (just a
> > > > HTTP
> > > > > > GET query) these endpoints to get a list of all the metrics
we
> > need:
> > > > > >
> > > > > > <host>:8074/jolokia/read/kafka.server:*
> > > > > > <host>:8074/jolokia/read/kafka.controller:*
> > > > > > <host>:8074/jolokia/read/kafka.log:*
> > > > > > <host>:8074/jolokia/read/kafka.network:*
> > > > > > <host>:8074/jolokia/read/java.lang:type=Memory
> > > > > >
> > > > > >
> > > > > > And that's it, we get a nice JSON that we can use the way we
want
> > :-)
> > > > > > I hope I didn't miss anything, but this should be it.
> > > > > >
> > > > > > M.
> > > > > >
> > > > > >
> > > > > > On 8 December 2017 at 09:28, Irtiza Ali <iali@an10.io>
wrote:
> > > > > >
> > > > > > > Hello Michal,
> > > > > > >
> > > > > > > Can you send me link to tutorial or provide some resources
for
> > the
> > > > > > Jolokia
> > > > > > > configuration with kafka>
> > > > > > >
> > > > > > >
> > > > > > > Thank you
> > > > > > > Irtiza
> > > > > > >
> > > > > > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski <
> > > > > > > michal.michalski@zalando.ie> wrote:
> > > > > > >
> > > > > > > > Hi Irtiza,
> > > > > > > >
> > > > > > > > We're using Jolokia and we had no problems with it.
> > > > > > > > It would be useful to know what exactly you did (how
you
> > "plugged
> > > > in"
> > > > > > > > Jolokia, how you configured it, what endpoint are
you
> querying
> > > > etc.)
> > > > > to
> > > > > > > > help you.
> > > > > > > >
> > > > > > > > On 6 December 2017 at 10:36, Irtiza Ali <iali@an10.io>
> wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > I am working python based Kafka monitoring application.
I
> am
> > > > unable
> > > > > > to
> > > > > > > > > figure out how to retrieve the metrics using
Jolokia. I
> have
> > > > enable
> > > > > > the
> > > > > > > > > port for metrics retrieval to 9999.
> > > > > > > > >
> > > > > > > > > I Have two questions
> > > > > > > > >
> > > > > > > > > 1) Is there something that I am not doing correctly.
> > > > > > > > > 2) Is there some other way to do it.
> > > > > > > > >
> > > > > > > > > With Regards
> > > > > > > > > Irtiza Alli
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message