kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sal...@gmail.com>
Subject Re: Number of Consumers Connected
Date Mon, 19 Jan 2015 23:58:41 GMT
Hi,

I use such tool

Consumer Offset Checker

Displays the:  Consumer Group, Topic, Partitions, Offset, logSize, Lag,
Owner for the specified set of Topics and Consumer Group
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker

To be able to know the consumer group, in zkCli.sh

[zk: localhost:2181(CONNECTED) 3] ls /
[transactional, admin, zookeeper, consumers, config, controller, storm,
brokers, controller_epoch]
[zk: localhost:2181(CONNECTED) 4] ls /consumers
[web-console-consumer-99295, web-console-consumer-37853,
web-console-consumer-30841, perf-consumer-92283, perf-consumer-21631,
perf-consumer-95281, perf-consumer-59296, web-console-consumer-52126,
web-console-consumer-89137, perf-consumer-72484, perf-consumer-80363,
web-console-consumer-47543, web-console-consumer-22509,
perf-consumer-16954, perf-consumer-53957, perf-consumer-39448,
web-console-consumer-17021, perf-consumer-88693,
web-console-consumer-48744, web-console-consumer-82543,
perf-consumer-89565, web-console-consumer-97959, perf-consumer-40427,
web-console-consumer-95350, web-console-consumer-26473,
web-console-consumer-79384, web-console-consumer-87777,
perf-consumer-91681, web-console-consumer-36136,
web-console-consumer-86924, perf-consumer-24510, perf-consumer-5888,
perf-consumer-73534, perf-consumer-92985, perf-consumer-7675,
perf-consumer-52306, perf-consumer-87352, web-console-consumer-30400]
[zk: localhost:2181(CONNECTED) 5]

I then run

root@exemplary-birds:/srv/kafka# bin/kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker --topic PofApiTest-1 --group
web-console-consumer-48744
Group           Topic                          Pid Offset
logSize         Lag             Owner
Exception in thread "main" org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /consumers/web-console-consumer-48744/offsets/PofApiTest-1/0
        at
org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
        at
org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
        at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
        at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
        at kafka.utils.ZkUtils$.readData(ZkUtils.scala:461)
        at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
        at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:89)
        at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
        at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
        at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:88)
        at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
        at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at
kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:152)
        at
kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/consumers/web-console-consumer-48744/offsets/PofApiTest-1/0
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
        at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
        at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
        at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
        at
org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
        ... 15 more

So consumer groups make confusing, I didn't specify the consumer-group-id
in producer, the only place I know to config group is consumer.properties


#consumer group id
group.id=test-consumer-group


Any hints? Thanks


AL






On Mon, Dec 15, 2014 at 6:46 PM, nitin sharma <kumarsharma.nitin@gmail.com>
wrote:

> got it ... thanks a lot.
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Mon, Dec 15, 2014 at 9:26 PM, Gwen Shapira <gshapira@cloudera.com>
> wrote:
> >
> > Hi Nitin,
> >
> > Go to where you installed zookeeper and run:
> >
> > bin/zkCli.sh -server 127.0.0.1:2181
> >
> > On Mon, Dec 15, 2014 at 6:09 PM, nitin sharma
> > <kumarsharma.nitin@gmail.com> wrote:
> > > Thanks Neha and Gwen for your responses..
> > >
> > > @Gwen -- Kindly explain how to perform the steps you have mentioned.
> how
> > > should i connect to a zookeeper..?
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Mon, Dec 15, 2014 at 6:36 PM, Neha Narkhede <neha@confluent.io>
> > wrote:
> > >>
> > >> In addition to Gwen's suggestion, we actually don't have jmx metrics
> > that
> > >> give you a list of actively consuming processes.
> > >>
> > >> On Mon, Dec 15, 2014 at 12:59 PM, Gwen Shapira <gshapira@cloudera.com
> >
> > >> wrote:
> > >> >
> > >> > Currently you can find the number of consumer groups through
> > ZooKeeper:
> > >> >
> > >> > connect to ZK and run
> > >> > ls /consumers
> > >> >
> > >> > and count the number of results
> > >> >
> > >> > On Mon, Dec 15, 2014 at 11:34 AM, nitin sharma
> > >> > <kumarsharma.nitin@gmail.com> wrote:
> > >> > > Hi Team,
> > >> > >
> > >> > > Is it possible to know how many Consumer Group connected to kafka
> > >> broker
> > >> > Ids
> > >> > > and as well as how many Instances within a Group are fetching
> > messages
> > >> > from
> > >> > > Kafka Brokers
> > >> > >
> > >> > > Regards,
> > >> > > Nitin Kumar Sharma.
> > >> >
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Neha
> > >>
> >
>



-- 

Alec Li

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message