storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junfeng Chen <k-2f...@hotmail.com>
Subject Re: KafkaSpout offset stored under wrong partition in zookeeper
Date Wed, 19 Oct 2016 09:41:50 GMT
Does the storm share the same zookeeper with Kafka? And where did you find
the error?


Regard,
Junfeng Chen

On Wed, Oct 19, 2016 at 5:30 PM, Daniccan VP <daniccan@iqsystech.com> wrote:

>
> String zookeeperConnectionString = "zookeeper-server01:2181,
> zookeeper-server02:2181,zookeeper-server03:2181,zookeeper-server04:2181";
>
> BrokerHosts hosts = new ZkHosts(zookeeperConnectionString);
>
> SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopicName, "/" +
> kafkaTopicName, kafkaSpoutConsumerGroupId);
>
> spoutConfig.bufferSizeBytes = 1024 * 1024 * 5;
> spoutConfig.fetchSizeBytes = 1024 * 1024 * 5;
> spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
> spoutConfig.useStartOffsetTimeIfOffsetOutOfRange = true;
>
> spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new
> CustomizedKeyValueScheme());
>
> KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
>
>
> Thanks and Regards,
> *Daniccan VP **|* *Junior Software Engineer*
> Email : daniccan@iqsystech.com
> ------------------------------
> *From:* darouwan@gmail.com [darouwan@gmail.com] on behalf of Junfeng Chen
> [k-2feng@hotmail.com]
> *Sent:* Wednesday, October 19, 2016 2:43 PM
> *To:* user@storm apache. org
> *Subject:* Re: KafkaSpout offset stored under wrong partition in zookeeper
>
> Can you provide your KafkaSpout configuration?
>
>
> Regard,
> Junfeng Chen
>
> On Wed, Oct 19, 2016 at 5:07 PM, Daniccan VP <daniccan@iqsystech.com>
> wrote:
>
>> Hi,
>>
>> I am running a Storm Topology with a KafkaSpout which consumes messages
>> from Kafka Brokers. The issue I am facing is that the assigned partition
>> for a worker process and the partition and offset information of the
>> consumer stored in zookeeper are different.
>>
>> For example, One of the worker processes is assigned with a partition id
>> of 1. But, it keeps searching for the last offset under partition 2 in
>> Zookeeper path. Has anyone else faced a similar issue ?
>>
>> Thanks and Regards,
>> *Daniccan VP **|* *Junior Software Engineer*
>> Email : daniccan@iqsystech.com
>> ************************************************************
>> ************************************************************
>> ******************************************* This email and any files
>> transmitted with it are confidential and intended solely for the use of the
>> individual or entity to whom they have been addressed. If you are not the
>> intended recipient, you are notified that disclosing, copying, distributing
>> or taking any action in reliance on the contents of this information is
>> strictly prohibited. Please notify the sender immediately by e-mail if you
>> have received this e-mail by mistake and delete this e-mail from your
>> system.
>>
>
> ************************************************************
> ************************************************************
> ******************************************* This email and any files
> transmitted with it are confidential and intended solely for the use of the
> individual or entity to whom they have been addressed. If you are not the
> intended recipient, you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system.
>

Mime
View raw message