storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniccan VP <danic...@iqsystech.com>
Subject RE: KafkaSpout offset stored under wrong partition in zookeeper
Date Wed, 19 Oct 2016 09:30:49 GMT

String zookeeperConnectionString = "zookeeper-server01:2181,zookeeper-server02:2181,zookeeper-server03:2181,zookeeper-server04:2181";

BrokerHosts hosts = new ZkHosts(zookeeperConnectionString);

SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopicName, "/" + kafkaTopicName, kafkaSpoutConsumerGroupId);

spoutConfig.bufferSizeBytes = 1024 * 1024 * 5;
spoutConfig.fetchSizeBytes = 1024 * 1024 * 5;
spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
spoutConfig.useStartOffsetTimeIfOffsetOutOfRange = true;

spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new CustomizedKeyValueScheme());

KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);


Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com
________________________________
From: darouwan@gmail.com [darouwan@gmail.com] on behalf of Junfeng Chen [k-2feng@hotmail.com]
Sent: Wednesday, October 19, 2016 2:43 PM
To: user@storm apache. org
Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper

Can you provide your KafkaSpout configuration?


Regard,
Junfeng Chen

On Wed, Oct 19, 2016 at 5:07 PM, Daniccan VP <daniccan@iqsystech.com<mailto:daniccan@iqsystech.com>>
wrote:
Hi,

I am running a Storm Topology with a KafkaSpout which consumes messages from Kafka Brokers.
The issue I am facing is that the assigned partition for a worker process and the partition
and offset information of the consumer stored in zookeeper are different.

For example, One of the worker processes is assigned with a partition id of 1. But, it keeps
searching for the last offset under partition 2 in Zookeeper path. Has anyone else faced a
similar issue ?

Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : daniccan@iqsystech.com<http://iqsystech.com>
*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they have been addressed. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from
your system.

*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended solely for the
use of the individual or entity to whom they have been addressed. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from
your system.

Mime
View raw message