kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: How to parse some of JMX Bean's names.
Date Fri, 20 Jun 2014 12:03:04 GMT
Hi Neha,

The patch is in https://issues.apache.org/jira/browse/KAFKA-1481 . It's
super-simple.  Does it require a review or can one submit it to Jenkins or
something else?

Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Wed, Jun 4, 2014 at 9:47 PM, Neha Narkhede <neha.narkhede@gmail.com>
wrote:

> Agree that 0.9 is a few months out. 0.8.2, on the other hand, is probably
> less than a month away. On one hand, we are trying our best to minimize the
> set of changes to the existing consumer in the interest of saving time to
> work on the new consumer. Since the old consumer is very complex and is now
> stable for some time, the motivation is to make fewer changes to it to
> maintain that stability. On the other hand, if there are critical bug
> fixes, it makes sense to patch the old consumer and do a point release.
>
> We would be happy to take a patch from you. How about we look at the size
> of the proposed changes and discuss a release timeline on the JIRA?
>
> Thanks,
> Neha
>
>
> On Wed, Jun 4, 2014 at 3:09 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > wrote:
>
> > Hi Guozhang,
> >
> > Since the new consumer is 2-3 months out ... hm, no, looks like 4 months
> > out - October -
> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> >
> > Any way you could change this in 0.8.1.2 or 0.8.2?  We can submit a
> patch.
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Wed, Jun 4, 2014 at 4:20 PM, Guozhang Wang <wangguoz@gmail.com>
> wrote:
> >
> > > I think we will make this change in the new consumer, which may be
> > released
> > > in 0.9.
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, Jun 4, 2014 at 12:13 PM, Vladimir Tretyakov <
> > > vladimir.tretyakov@sematext.com> wrote:
> > >
> > > > Hi, thx Guozhang Wang, looking forward.
> > > >
> > > > When do you think this changes will available? 0.8.2? Jul 2014
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> > ?
> > > Or
> > > > later?
> > > >
> > > > Best regards, Vladimir.
> > > >
> > > >
> > > > On Wed, Jun 4, 2014 at 8:01 PM, Guozhang Wang <wangguoz@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Vladimir, comments in-lined.
> > > > >
> > > > >
> > > > > On Wed, Jun 4, 2014 at 6:08 AM, Vladimir Tretyakov <
> > > > > vladimir.tretyakov@sematext.com> wrote:
> > > > >
> > > > > > Hi again, few more questions from me:
> > > > > >
> > > > > > *1.*
> > > > > >
> > > > > > What I see in JMX:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
> > > > > >
> > > > > > From code:
> > > > > >
> > > > > > newGauge(
> > > > > >         config.clientId + "-" + config.groupId + "-" +
> > > > topicThreadId._1 +
> > > > > > "-" + topicThreadId._2 + "-FetchQueueSize",
> > > > > >         new Gauge[Int] {
> > > > > >           def value = q.size
> > > > > >         }
> > > > > >       )
> > > > > >
> > > > > > I've tried to parse part as I've understood they.
> > > > > >
> > > > > > config.clientId       >> af_servers
> > > > > > topicThreadId._1 >> af_servers-spm_new_cluster_topic
> > > > > > topicThreadId._2 >>
> > > af_servers_wawanawna-Dell-1401353748289-fcaaea29-0
> > > > > >
> > > > > > Yes I can suppose that this topicThreadId._1 will always looks
> like
> > > > > > GROUP_ID+TOPIC and topicThreadId._2 will contain CONSUMER HOST,
> but
> > > > will
> > > > > it
> > > > > > always true?
> > > > >
> > > > >
> > > > >
> > > > > With the new consumer coming soon, the metrics naming schemes would
> > > very
> > > > > likely to be refined. So I cannot say this will always be true.
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > *2.*
> > > > > >
> > > > > > From code I see that sometimes Kafka uses "_" as separator,
not
> > only
> > > > "-":
> > > > > >
> > > > > > val consumerIdString = {
> > > > > >     var consumerUuid : String = null
> > > > > >     config.consumerId match {
> > > > > >       case Some(consumerId) // for testing only
> > > > > >       => consumerUuid = consumerId
> > > > > >       case None // generate unique consumerId automatically
> > > > > >       => val uuid = UUID.randomUUID()
> > > > > >       consumerUuid = "%s-%d-%s".format(
> > > > > >         InetAddress.getLocalHost.getHostName,
> > > System.currentTimeMillis,
> > > > > >         uuid.getMostSignificantBits().toHexString.substring(0,8))
> > > > > >     }
> > > > > >     config.groupId + "_" + consumerUuid
> > > > > >   }
> > > > > >
> > > > > > That means if user will use "_" as part of his host/topic/groupId
> > > name
> > > > it
> > > > > > maybe be a problem to parse string like:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
> > > > > >
> > > > > > Look at part: "spm_new_cluster_topic-af_servers_wawanawna-Dell",
> > what
> > > > is
> > > > > > host name here "servers_wawanawna-Dell" or "wawanawna-Dell"
?
> > > > > >
> > > > > > So from one side if we want to be able parse name without any
> > > problems
> > > > we
> > > > > > have to avoid using "-" and "_" in host/topic/groupId/clientId,
> but
> > > at
> > > > > the
> > > > > > same time I see (from
> > > > > >
> > http://grokbase.com/t/kafka/users/133xfsnpdh/cant-use-in-client-name
> > > ):
> > > > > >
> > > > > > *"Client id is used for registering jmx beans for monitoring.
> > Because
> > > > of
> > > > > > the*
> > > > > > *restrictions in bean names, we limit the client id to be only
> > > > > > alpha-numeric*
> > > > > > *plus "-" and "_"."*
> > > > > >
> > > > > > Does that mean user can use only camelCase in his
> > > > > > host/topic/groupId/clientId for distinguish one part of name
from
> > > > > another?
> > > > > >
> > > > > > Is this a problem? Or I didn't understand something?
> > > > > >
> > > > >
> > > > > Yeah I agree this is a problem, and we should fix it in the new
> > > consumer.
> > > > >
> > > > >
> > > > > >
> > > > > > Best regards from Sematext.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 3, 2014 at 3:24 AM, Otis Gospodnetic <
> > > > > > otis.gospodnetic@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Guozhang,
> > > > > > >
> > > > > > > On Mon, Jun 2, 2014 at 7:18 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > That is indeed a problem, for now, we recommend group
name
> and
> > > > topic
> > > > > > > names
> > > > > > > > to use "_" when there is a need for "-", but this
should be
> > fixed
> > > > > > > > systematically.
> > > > > > > >
> > > > > > >
> > > > > > > Right!
> > > > > > >
> > > > > > > For you use case, could you change your topic/group name
using
> > "_"?
> > > > > > >
> > > > > > >
> > > > > > > Our own Kafka doesn't use topics with "-" characters, so
we
> don't
> > > > have
> > > > > a
> > > > > > > problem.
> > > > > > >
> > > > > > > The problem, in our case, is that we have a general (Kafka)
> > > > monitoring
> > > > > > tool
> > > > > > > that other people use to monitor Kafka - see
> > > > http://sematext.com/spm/
> > > > > .
> > > > > > >  So
> > > > > > > we can't really tell people "hey, our tool will work but
only
> if
> > > you
> > > > > > don't
> > > > > > > have a dash in your topic names and hosts and ... because
if
> you
> > > use
> > > > > > dashes
> > > > > > > we won't know how to parse your Kafka's MBean names" :)
> > > > > > >
> > > > > > >
> > > > > > > > Also, do you mind to file a JIRA ticket to keep track
of this
> > > > issue?
> > > > > > >
> > > > > > >
> > > > > > > Here it is: https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > > >
> > > > > > > Otis
> > > > > > > --
> > > > > > > Performance Monitoring * Log Analytics * Search Analytics
> > > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jun 2, 2014 at 5:18 AM, Vladimir Tretyakov
<
> > > > > > > > vladimir.tretyakov@sematext.com> wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > We are adding Kafka 0.8.x monitoring support
to SPM
> > > > > > > > > <http://sematext.com/spm/> here at Sematext.
> Unfortunately,
> > we
> > > > > > quickly
> > > > > > > > hit
> > > > > > > > > an issue caused by the new bean naming convention
that
> embeds
> > > > > things
> > > > > > > like
> > > > > > > > > topic and host names in the beans along with
metrics,
> > separated
> > > > by
> > > > > > > > dashes,
> > > > > > > > > making it hard to parse these beans.
> > > > > > > > >
> > > > > > > > > To put it simply: it is hard/impossible to automatically
> > figure
> > > > out
> > > > > > > which
> > > > > > > > > part of the bean name is e.g. consumer group,
which is the
> > > topic,
> > > > > > which
> > > > > > > > is
> > > > > > > > > the host name, and which is the name of the metric.
> > > > > > > > >
> > > > > > > > > Let me show you what I mean:
> > > > > > > > >
> > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > >
> > > > > > > > >
> >  name="af_servers-spm_topic-BytesPerSec"
> > > > > > > > >
> > > > > > > > > Here we actually CAN extract:
> > > > > > > > >
> > > > > > > > >  * consumer group ('af_servers')
> > > > > > > > >
> > > > > > > > >  * topic ('spm_topic')
> > > > > > > > >
> > > > > > > > >  * metric (‘BytesPerSec’)
> > > > > > > > >
> > > > > > > > > BUT what if the consumer group id and/or topic
name contain
> > > '-'?
> > > > > > > > >
> > > > > > > > > Then how would we extract consumer group and
topic?
> > > > > > > > >
> > > > > > > > > Here is a concrete example of this problem:
> > > > > > > > >
> > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > > >
> > > > > > > > >
> name="af-servers-spm-topic-BytesPerSec"
> > > > > > > > >
> > > > > > > > > How can we know what is group id or topic name
here?
> > > > > > > > >
> > > > > > > > > This looks like a problem to me, but maybe I’m
missing
> > > something?
> > > > > > > > >
> > > > > > > > > Is it possible to have all these values (group
id, topic
> > name)
> > > as
> > > > > > > > separate
> > > > > > > > > attributes inside JMX bean?
> > > > > > > > >
> > > > > > > > > Or maybe the problem could be solved if a different
> delimiter
> > > was
> > > > > > used,
> > > > > > > > > such as the pipe (“I”)?
> > > > > > > > >
> > > > > > > > > It is really needed things and will be nice to
have it to
> > build
> > > > > good
> > > > > > > tool
> > > > > > > > > for monitoring.
> > > > > > > > >
> > > > > > > > > Thx and best regards from Sematext.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message