kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Michalski <michal.michal...@zalando.ie>
Subject Re: Kafka Monitoring
Date Fri, 08 Dec 2017 15:19:00 GMT
Hi,

We have no modifications in that file - what we do is having a "wrapper"
that's just a Docker "entrypoint" (just a bash script) which contents is:

export KAFKA_OPTS="$KAFKA_OPTS
-javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0"
exec ${KAFKA_DIR}/bin/kafka-server-start.sh
${KAFKA_DIR}/config/server.properties

Regarding your second question - we have an in-house monitoring solution
that allows querying that endpoint and extracting metrics from the JSON
returned.

If you're asking about what you should monitor, I think these links will
answer your question:
https://docs.confluent.io/3.0.0/kafka/monitoring.html
https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/


On 8 December 2017 at 14:56, Irtiza Ali <iali@an10.io> wrote:

> Thanks Michal, can you kindly send me you kafka-run-class.sh and
> kafka-server-start.sh file, I have look what have you done. Because I have
> done same thing that you explained above but when i do this <
> http://localhost:9999/jolokia/list> i get only metrics for the zookeeper
> but not the above metrics.
>
> How are you using Jolokia for monitoring kafka cluster??
>
> Thanks in advance
>
> On Fri, Dec 8, 2017 at 3:10 PM, Michal Michalski <
> michal.michalski@zalando.ie> wrote:
>
> > Hi Irtiza,
> >
> > I don't have any tutorial, but I can tell you what we do :-)
> >
> > First of all we have Jolokia agent jar included in our Kafka Docker
> image.
> > Then we append this to KAFKA_OPTS
> >
> > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0
> >
> >
> > Relevant port is then "exposed" in Docker image and "allowed" in AWS
> > Security Group.
> > Then our monitoring tool is querying the following endpoins (just a HTTP
> > GET query) these endpoints to get a list of all the metrics we need:
> >
> > <host>:8074/jolokia/read/kafka.server:*
> > <host>:8074/jolokia/read/kafka.controller:*
> > <host>:8074/jolokia/read/kafka.log:*
> > <host>:8074/jolokia/read/kafka.network:*
> > <host>:8074/jolokia/read/java.lang:type=Memory
> >
> >
> > And that's it, we get a nice JSON that we can use the way we want :-)
> > I hope I didn't miss anything, but this should be it.
> >
> > M.
> >
> >
> > On 8 December 2017 at 09:28, Irtiza Ali <iali@an10.io> wrote:
> >
> > > Hello Michal,
> > >
> > > Can you send me link to tutorial or provide some resources for the
> > Jolokia
> > > configuration with kafka>
> > >
> > >
> > > Thank you
> > > Irtiza
> > >
> > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski <
> > > michal.michalski@zalando.ie> wrote:
> > >
> > > > Hi Irtiza,
> > > >
> > > > We're using Jolokia and we had no problems with it.
> > > > It would be useful to know what exactly you did (how you "plugged in"
> > > > Jolokia, how you configured it, what endpoint are you querying etc.)
> to
> > > > help you.
> > > >
> > > > On 6 December 2017 at 10:36, Irtiza Ali <iali@an10.io> wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I am working python based Kafka monitoring application. I am unable
> > to
> > > > > figure out how to retrieve the metrics using Jolokia. I have enable
> > the
> > > > > port for metrics retrieval to 9999.
> > > > >
> > > > > I Have two questions
> > > > >
> > > > > 1) Is there something that I am not doing correctly.
> > > > > 2) Is there some other way to do it.
> > > > >
> > > > > With Regards
> > > > > Irtiza Alli
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message