storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniccan VP <danic...@iqsystech.com>
Subject RE: KafkaSpout offset stored under wrong partition in zookeeper
Date Wed, 19 Oct 2016 09:54:38 GMT
Yes. Storm and Kafka both share the same zookeeper ensemble.

I found the following error in my worker.log file,

2016-10-19 09:21:04.326 o.a.s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
GlobalPartitionInformation{topic=mytopic, partitionMap={0=kafka-broker01:9092, 1=kafka-broker02:9092,
2=kafka-broker03:9092}}
2016-10-19 09:21:04.330 o.a.s.k.KafkaUtils [INFO] Task [2/3] assigned [Partition{host=kafka-broker01:9092,
topic=mytopic, partition=1}]
2016-10-19 09:21:04.330 o.a.s.k.ZkCoordinator [INFO] Task [2/3] Deleted partition managers:
[]
2016-10-19 09:21:04.330 o.a.s.k.ZkCoordinator [INFO] Task [2/3] New partition managers: [Partition{host=kafka-broker01:9092,
topic=mytopic, partition=1}]
2016-10-19 09:21:04.872 o.a.s.k.PartitionManager [INFO] Read partition information from: /mytopic/myconsumergroup/partition_1
 --> null
2016-10-19 09:21:05.006 o.a.s.k.PartitionManager [INFO] No partition information found, using
configuration to determine offset
2016-10-19 09:21:05.007 o.a.s.k.PartitionManager [INFO] Last commit offset from zookeeper:
0

And In my Zookeeper Client, I could find only the partition_2 file in the zookeeper path,

[zk: localhost:2181(CONNECTED) 6] ls /mytopic/myconsumergroup
[partition_2]

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com
________________________________
From: darouwan@gmail.com [darouwan@gmail.com] on behalf of Junfeng Chen [k-2feng@hotmail.com]
Sent: Wednesday, October 19, 2016 3:11 PM
To: user@storm apache. org
Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper

Does the storm share the same zookeeper with Kafka? And where did you find the error?


Regard,
Junfeng Chen

On Wed, Oct 19, 2016 at 5:30 PM, Daniccan VP <daniccan@iqsystech.com<mailto:daniccan@iqsystech.com>>
wrote:

String zookeeperConnectionString = "zookeeper-server01:2181,zookeeper-server02:2181,zookeeper-server03:2181,zookeeper-server04:2181";

BrokerHosts hosts = new ZkHosts(zookeeperConnectionString);

SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopicName, "/" + kafkaTopicName, kafkaSpoutConsumerGroupId);

spoutConfig.bufferSizeBytes = 1024 * 1024 * 5;
spoutConfig.fetchSizeBytes = 1024 * 1024 * 5;
spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
spoutConfig.useStartOffsetTimeIfOffsetOutOfRange = true;

spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new CustomizedKeyValueScheme());

KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);


Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com<http://iqsystech.com>
________________________________
From: darouwan@gmail.com<mailto:darouwan@gmail.com> [darouwan@gmail.com<mailto:darouwan@gmail.com>]
on behalf of Junfeng Chen [k-2feng@hotmail.com<mailto:k-2feng@hotmail.com>]
Sent: Wednesday, October 19, 2016 2:43 PM
To: user@storm apache. org
Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper

Can you provide your KafkaSpout configuration?


Regard,
Junfeng Chen

On Wed, Oct 19, 2016 at 5:07 PM, Daniccan VP <daniccan@iqsystech.com<mailto:daniccan@iqsystech.com>>
wrote:
Hi,

I am running a Storm Topology with a KafkaSpout which consumes messages from Kafka Brokers.
The issue I am facing is that the assigned partition for a worker process and the partition
and offset information of the consumer stored in zookeeper are different.

For example, One of the worker processes is assigned with a partition id of 1. But, it keeps
searching for the last offset under partition 2 in Zookeeper path. Has anyone else faced a
similar issue ?

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com<http://iqsystech.com>
*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they have been addressed. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from
your system.

*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they have been addressed. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from
your system.

*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they have been addressed. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from
your system.

Mime
View raw message