kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorit Hagedorn <jhaged...@plusline.net>
Subject Re: Kafka Cluster Consumer - Failover not working
Date Wed, 13 Feb 2019 11:30:19 GMT
Hi,

that did the trick, thanks!


On 2/13/19 12:09 PM, M. Manna wrote:
> In that case do let us know if it worked for you.
>
> On Wed, 13 Feb 2019 at 11:07, Jorit Hagedorn <jhagedorn@plusline.net> wrote:
>
>> Hi,
>>
>> I did not know about this configuration file. Content:
>>
>> bootstrap.servers=localhost:9092
>> group.id=test-consumer-group
>>
>> Think I know what to do now.
>>
>> Kind Regards
>>
>> Jorit
>>
>> On 2/13/19 11:38 AM, M. Manna wrote:
>>> What is your consumer prop file ? How have you adjusted the properties?
>>>
>>>
>>>
>>> On Wed, 13 Feb 2019 at 09:41, Jorit Hagedorn <jhagedorn@plusline.net>
>> wrote:
>>>> Hello,
>>>>
>>>> I've setup a Kafka- / Zookeeper-Cluster with 3 vms:
>>>>
>>>> 101
>>>> 201
>>>> 102
>>>>
>>>> These 3 servers run as cluster. Configs below:
>>>>
>>>> Kafka:
>>>>
>>>> broker.id=101  #  102 and 201
>>>> num.network.threads=3
>>>> num.io.threads=8
>>>> socket.send.buffer.bytes=102400
>>>> socket.receive.buffer.bytes=102400
>>>> socket.request.max.bytes=104857600
>>>> log.dirs=/opt/kafka/kafka-data
>>>> num.partitions=1
>>>> num.recovery.threads.per.data.dir=1
>>>> offsets.topic.replication.factor=1
>>>> transaction.state.log.replication.factor=1
>>>> transaction.state.log.min.isr=1
>>>> log.retention.hours=168
>>>> log.segment.bytes=1073741824
>>>> log.retention.check.interval.ms=300000
>>>> zookeeper.connect=10.1.221.13:2181,10.2.172.13:2181,10.1.221.16:2181
>>>> zookeeper.connection.timeout.ms=6000
>>>> group.initial.rebalance.delay.ms=0
>>>>
>>>>
>>>> Zookeeper:
>>>>
>>>> dataDir=/opt/kafka/zookeeper-data
>>>> clientPort=2181
>>>> initLimit=10
>>>> syncLimit=5
>>>> maxClientCnxns=0
>>>>
>>>> server.101=10.1.221.13:2888:3888
>>>> server.201=10.2.172.13:2888:3888
>>>> server.102=10.1.221.16:2888:3888
>>>>
>>>>
>>>> Full cluster is available:
>>>>
>>>>
>>>> /opt/kafka/bin/zookeeper-shell.sh localhost:2181 <<< "ls /brokers/ids"
>>>> Connecting to localhost:2181
>>>> Welcome to ZooKeeper!
>>>> JLine support is disabled
>>>>
>>>> WATCHER::
>>>>
>>>> WatchedEvent state:SyncConnected type:None path:null
>>>> [101, 102, 201]
>>>>
>>>> After that, I've created a replicated topic:
>>>>
>>>> /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181
>>>> --replication-factor 3 --partitions 1 --topic myreplicatedtopic
>>>>
>>>> Verify:
>>>>
>>>> /opt/kafka/bin/kafka-topics.sh --describe --topic myreplicatedtopic
>>>> --zookeeper localhost:2181
>>>> Topic:myreplicatedtopic    PartitionCount:1    ReplicationFactor:3
>>>> Configs:
>>>>     Topic: myreplicatedtopic    Partition: 0    Leader: 102    Replicas:
>>>> 102,101,201    Isr: 102,101,201
>>>>
>>>>
>>>> The issue is that a consumer which connects to node 102 for example will
>>>> stop working and not use any of the other 2 servers for failover.
>>>> As a consumer we currently have logstash. The consumer seems to fail
>>>> once the first server that started in the cluster becomes unavailable
>>>> for some reason.
>>>>
>>>> Producing messages (with filebeat) is always working, as long as the
>>>> majority of the servers are up, which is the expected behaviour.
>>>>
>>>> What am I missing here?
>>>>
>>>> Kind Regards
>>>>
>>>> Jorit
>>>>
>>>>
>>>>



Mime
View raw message