storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elyahou Ittah <elyaho...@fiverr.com>
Subject Re: KafkaSpout reconsume all kafka messages after kafka rolling restart
Date Mon, 21 Aug 2017 20:20:48 GMT
Hi Stig,

I don't have this kind of errors normally. It just occured at the rolling
restart of kafka.

Also the __consumer_offsets configuration is:
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3
Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 2,1,0 Isr: 0,1
...

The fact that the replication factor is 3 even if there is only two broker
can cause an issue ?


On Mon, Aug 21, 2017 at 6:51 PM, Stig Rohde Døssing <srdo@apache.org> wrote:

> The spout will reemit some messages if it fails to commit offsets to
> Kafka. Are these CommitFailedExceptions occuring in your logs normally?
>
> Also since the spout stores offsets in Kafka, you may want to check the
> replication factor on that topic by running `./kafka-topics.sh --zookeeper
> localhost:2181 --describe --topic __consumer_offsets` in one of your Kafka
> /bin directories.
>
> 2017-08-21 17:17 GMT+02:00 Elyahou Ittah <elyahou.i@fiverr.com>:
>
>> The config is the default one, I just set the bootstrap server.
>>
>> Kafka version is 0.11
>>
>> Storm-kafka-client is 1.1.0
>>
>> On Mon, Aug 21, 2017 at 5:48 PM, Stig Rohde Døssing <srdo@apache.org>
>> wrote:
>>
>>> Hi Elyahou,
>>>
>>> Could you post your spout configuration, Kafka version and
>>> storm-kafka-client version? The logs imply that your spouts are not polling
>>> often enough.
>>>
>>> 2017-08-21 <20%2017%2008%2021> 9:44 GMT+02:00 Elyahou Ittah <
>>> elyahou.i@fiverr.com>:
>>>
>>>> I noticed that storm kafka spout reconsume all kafka message after a
>>>> rolling restart of kafka cluster.
>>>>
>>>> This issue occured only with kafkaSpout consumer and not for my other
>>>> consumers (ruby based  using the kafka consumer API like kafkaSpout)
>>>>
>>>> Attached logs of the spout.
>>>>
>>>> Do you know what can cause this kind of behavior ?
>>>>
>>>
>>>
>>
>

Mime
View raw message