kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: How to parse some of JMX Bean's names.
Date Thu, 05 Jun 2014 01:47:22 GMT
Agree that 0.9 is a few months out. 0.8.2, on the other hand, is probably
less than a month away. On one hand, we are trying our best to minimize the
set of changes to the existing consumer in the interest of saving time to
work on the new consumer. Since the old consumer is very complex and is now
stable for some time, the motivation is to make fewer changes to it to
maintain that stability. On the other hand, if there are critical bug
fixes, it makes sense to patch the old consumer and do a point release.

We would be happy to take a patch from you. How about we look at the size
of the proposed changes and discuss a release timeline on the JIRA?

Thanks,
Neha


On Wed, Jun 4, 2014 at 3:09 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:

> Hi Guozhang,
>
> Since the new consumer is 2-3 months out ... hm, no, looks like 4 months
> out - October -
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>
> Any way you could change this in 0.8.1.2 or 0.8.2?  We can submit a patch.
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Wed, Jun 4, 2014 at 4:20 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > I think we will make this change in the new consumer, which may be
> released
> > in 0.9.
> >
> > Guozhang
> >
> >
> > On Wed, Jun 4, 2014 at 12:13 PM, Vladimir Tretyakov <
> > vladimir.tretyakov@sematext.com> wrote:
> >
> > > Hi, thx Guozhang Wang, looking forward.
> > >
> > > When do you think this changes will available? 0.8.2? Jul 2014
> > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> ?
> > Or
> > > later?
> > >
> > > Best regards, Vladimir.
> > >
> > >
> > > On Wed, Jun 4, 2014 at 8:01 PM, Guozhang Wang <wangguoz@gmail.com>
> > wrote:
> > >
> > > > Hello Vladimir, comments in-lined.
> > > >
> > > >
> > > > On Wed, Jun 4, 2014 at 6:08 AM, Vladimir Tretyakov <
> > > > vladimir.tretyakov@sematext.com> wrote:
> > > >
> > > > > Hi again, few more questions from me:
> > > > >
> > > > > *1.*
> > > > >
> > > > > What I see in JMX:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
> > > > >
> > > > > From code:
> > > > >
> > > > > newGauge(
> > > > >         config.clientId + "-" + config.groupId + "-" +
> > > topicThreadId._1 +
> > > > > "-" + topicThreadId._2 + "-FetchQueueSize",
> > > > >         new Gauge[Int] {
> > > > >           def value = q.size
> > > > >         }
> > > > >       )
> > > > >
> > > > > I've tried to parse part as I've understood they.
> > > > >
> > > > > config.clientId       >> af_servers
> > > > > topicThreadId._1 >> af_servers-spm_new_cluster_topic
> > > > > topicThreadId._2 >>
> > af_servers_wawanawna-Dell-1401353748289-fcaaea29-0
> > > > >
> > > > > Yes I can suppose that this topicThreadId._1 will always looks like
> > > > > GROUP_ID+TOPIC and topicThreadId._2 will contain CONSUMER HOST, but
> > > will
> > > > it
> > > > > always true?
> > > >
> > > >
> > > >
> > > > With the new consumer coming soon, the metrics naming schemes would
> > very
> > > > likely to be refined. So I cannot say this will always be true.
> > > >
> > > >
> > > >
> > > > >
> > > > > *2.*
> > > > >
> > > > > From code I see that sometimes Kafka uses "_" as separator, not
> only
> > > "-":
> > > > >
> > > > > val consumerIdString = {
> > > > >     var consumerUuid : String = null
> > > > >     config.consumerId match {
> > > > >       case Some(consumerId) // for testing only
> > > > >       => consumerUuid = consumerId
> > > > >       case None // generate unique consumerId automatically
> > > > >       => val uuid = UUID.randomUUID()
> > > > >       consumerUuid = "%s-%d-%s".format(
> > > > >         InetAddress.getLocalHost.getHostName,
> > System.currentTimeMillis,
> > > > >         uuid.getMostSignificantBits().toHexString.substring(0,8))
> > > > >     }
> > > > >     config.groupId + "_" + consumerUuid
> > > > >   }
> > > > >
> > > > > That means if user will use "_" as part of his host/topic/groupId
> > name
> > > it
> > > > > maybe be a problem to parse string like:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
> > > > >
> > > > > Look at part: "spm_new_cluster_topic-af_servers_wawanawna-Dell",
> what
> > > is
> > > > > host name here "servers_wawanawna-Dell" or "wawanawna-Dell" ?
> > > > >
> > > > > So from one side if we want to be able parse name without any
> > problems
> > > we
> > > > > have to avoid using "-" and "_" in host/topic/groupId/clientId, but
> > at
> > > > the
> > > > > same time I see (from
> > > > >
> http://grokbase.com/t/kafka/users/133xfsnpdh/cant-use-in-client-name
> > ):
> > > > >
> > > > > *"Client id is used for registering jmx beans for monitoring.
> Because
> > > of
> > > > > the*
> > > > > *restrictions in bean names, we limit the client id to be only
> > > > > alpha-numeric*
> > > > > *plus "-" and "_"."*
> > > > >
> > > > > Does that mean user can use only camelCase in his
> > > > > host/topic/groupId/clientId for distinguish one part of name from
> > > > another?
> > > > >
> > > > > Is this a problem? Or I didn't understand something?
> > > > >
> > > >
> > > > Yeah I agree this is a problem, and we should fix it in the new
> > consumer.
> > > >
> > > >
> > > > >
> > > > > Best regards from Sematext.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 3, 2014 at 3:24 AM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Guozhang,
> > > > > >
> > > > > > On Mon, Jun 2, 2014 at 7:18 PM, Guozhang Wang <
> wangguoz@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > That is indeed a problem, for now, we recommend group name
and
> > > topic
> > > > > > names
> > > > > > > to use "_" when there is a need for "-", but this should
be
> fixed
> > > > > > > systematically.
> > > > > > >
> > > > > >
> > > > > > Right!
> > > > > >
> > > > > > For you use case, could you change your topic/group name using
> "_"?
> > > > > >
> > > > > >
> > > > > > Our own Kafka doesn't use topics with "-" characters, so we
don't
> > > have
> > > > a
> > > > > > problem.
> > > > > >
> > > > > > The problem, in our case, is that we have a general (Kafka)
> > > monitoring
> > > > > tool
> > > > > > that other people use to monitor Kafka - see
> > > http://sematext.com/spm/
> > > > .
> > > > > >  So
> > > > > > we can't really tell people "hey, our tool will work but only
if
> > you
> > > > > don't
> > > > > > have a dash in your topic names and hosts and ... because if
you
> > use
> > > > > dashes
> > > > > > we won't know how to parse your Kafka's MBean names" :)
> > > > > >
> > > > > >
> > > > > > > Also, do you mind to file a JIRA ticket to keep track of
this
> > > issue?
> > > > > >
> > > > > >
> > > > > > Here it is: https://issues.apache.org/jira/browse/KAFKA-1481
> > > > > >
> > > > > > Otis
> > > > > > --
> > > > > > Performance Monitoring * Log Analytics * Search Analytics
> > > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 2, 2014 at 5:18 AM, Vladimir Tretyakov <
> > > > > > > vladimir.tretyakov@sematext.com> wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > We are adding Kafka 0.8.x monitoring support to SPM
> > > > > > > > <http://sematext.com/spm/> here at Sematext.
Unfortunately,
> we
> > > > > quickly
> > > > > > > hit
> > > > > > > > an issue caused by the new bean naming convention
that embeds
> > > > things
> > > > > > like
> > > > > > > > topic and host names in the beans along with metrics,
> separated
> > > by
> > > > > > > dashes,
> > > > > > > > making it hard to parse these beans.
> > > > > > > >
> > > > > > > > To put it simply: it is hard/impossible to automatically
> figure
> > > out
> > > > > > which
> > > > > > > > part of the bean name is e.g. consumer group, which
is the
> > topic,
> > > > > which
> > > > > > > is
> > > > > > > > the host name, and which is the name of the metric.
> > > > > > > >
> > > > > > > > Let me show you what I mean:
> > > > > > > >
> > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > >
> > > > > > > >
>  name="af_servers-spm_topic-BytesPerSec"
> > > > > > > >
> > > > > > > > Here we actually CAN extract:
> > > > > > > >
> > > > > > > >  * consumer group ('af_servers')
> > > > > > > >
> > > > > > > >  * topic ('spm_topic')
> > > > > > > >
> > > > > > > >  * metric (‘BytesPerSec’)
> > > > > > > >
> > > > > > > > BUT what if the consumer group id and/or topic name
contain
> > '-'?
> > > > > > > >
> > > > > > > > Then how would we extract consumer group and topic?
> > > > > > > >
> > > > > > > > Here is a concrete example of this problem:
> > > > > > > >
> > > > > > > > kafka.consumer:type="ConsumerTopicMetrics",
> > > > > > > >
> > > > > > > >                       name="af-servers-spm-topic-BytesPerSec"
> > > > > > > >
> > > > > > > > How can we know what is group id or topic name here?
> > > > > > > >
> > > > > > > > This looks like a problem to me, but maybe I’m missing
> > something?
> > > > > > > >
> > > > > > > > Is it possible to have all these values (group id,
topic
> name)
> > as
> > > > > > > separate
> > > > > > > > attributes inside JMX bean?
> > > > > > > >
> > > > > > > > Or maybe the problem could be solved if a different
delimiter
> > was
> > > > > used,
> > > > > > > > such as the pipe (“I”)?
> > > > > > > >
> > > > > > > > It is really needed things and will be nice to have
it to
> build
> > > > good
> > > > > > tool
> > > > > > > > for monitoring.
> > > > > > > >
> > > > > > > > Thx and best regards from Sematext.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message