kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: kafka availability
Date Thu, 03 Apr 2014 03:41:02 GMT
bin/kafka-topics.sh --zookeeper zookeeper1 --topic test.app4 --describe
Topic:test.app4    PartitionCount:1    ReplicationFactor:2    Configs:
    Topic: test.app4    Partition: 0    Leader: 1    Replicas: 1,0    Isr: 1

Is broker 0 up when you did the above? Normally, when both brokers are up,
isr should include both.

Thanks,

Jun


On Wed, Apr 2, 2014 at 5:53 AM, Anatoly Deyneka <adeyneka@gmail.com> wrote:

> Hi,
>
> I perform availability tests on next kafka setup:
> - 3 nodes(2f+1) for zookeeper(zookeeper1, zookeeper2, zookeeper3)
> - 2 nodes(f+1) for kafka brokers(id:0,host:kafka.broker1,port:9092;
> id:1,host:kafka.broker2,port:9092)
> I use java producer and console consumer.
>
> Test stepts:
> 1. create topic test.app4
>
> bin/kafka-topics.sh --zookeeper zookeeper1 --create --replication-factor 2
> --partitions 1 --topic test.app4
>
> bin/kafka-topics.sh --zookeeper zookeeper1 --topic test.app4 --describe
> Topic:test.app4    PartitionCount:1    ReplicationFactor:2    Configs:
>     Topic: test.app4    Partition: 0    Leader: 1    Replicas: 1,0    Isr:
> 1
>
> 2. shutdown kafka.broker2(the leader for topic), kafka.broker1 is alive
> --> consumer fails in infinite loop:
>
> WARN Fetching topic metadata with correlation id 13 for topics
> [Set(test.app4)] from broker [id:0,host:kafka.broker1,port:9092] failed
> (kafka.client.ClientUtils$)
> java.net.ConnectException: Connection refused
> ...
> WARN
>
> [console-consumer-95116_ad-laptop-1396433768258-bb700f4c-leader-finder-thread],
> Failed to find leader for Set([test.app4,0])
> (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
> kafka.common.KafkaException: fetching topic metadata for topics
> [Set(test.app4)] from broker
> [ArrayBuffer(id:0,host:kafka.broker1,port:9092)] failed
> ...
> Caused by: java.net.ConnectException: Connection refused
> ...
> ERROR Producer connection to kafka.broker1:9092 unsuccessful
> (kafka.producer.SyncProducer)
> java.net.ConnectException: Connection refused
> ...
>
> bin/kafka-topics.sh --zookeeper zookeeper1 --topic test.app4
> --describeTopic:test.app4    PartitionCount:1    ReplicationFactor:2
> Configs:
>     Topic: test.app4    Partition: 0    Leader: 0    Replicas: 1,0    Isr:
> 0
>
> The java producer error:
> INFO  [kafka.producer.async.DefaultEventHandler] Back off for 100 ms before
> retrying send. Remaining retries = 2
> INFO  [kafka.client.ClientUtils$] Fetching metadata from broker
> id:0,host:kafka.broker1,port:9092 with correlation id 236 for 1 topic(s)
> Set(test.app4)
> ERROR [kafka.producer.SyncProducer] Producer connection to
> kafka.broker1:9092 unsuccessful
> java.net.ConnectException: Connection refused
>     ...
> WARN  [kafka.client.ClientUtils$] Fetching topic metadata with correlation
> id 236 for topics [Set(test.app4)] from broker
> [id:0,host:kafka.broker1,port:9092] failed
> java.net.ConnectException: Connection refused
>     ...
> INFO  [kafka.client.ClientUtils$] Fetching metadata from broker
> id:1,host:kafka.broker2,port:9092 with correlation id 236 for 1 topic(s)
> Set(test.app4)
> ERROR [kafka.producer.SyncProducer] Producer connection to
> kafka.broker2:9092 unsuccessful
> java.net.ConnectException: Connection refused
>    ...
> WARN  [kafka.client.ClientUtils$] Fetching topic metadata with correlation
> id 236 for topics [Set(test.app4)] from broker
> [id:1,host:kafka.broker2,port:9092] failed
> java.net.ConnectException: Connection refused
>     ...
> ERROR [kafka.utils.Utils$] fetching topic metadata for topics
> [Set(test.app4)] from broker
> [ArrayBuffer(id:0,host:kafka.broker1,port:9092,
> id:1,host:kafka.broker2,port:9092)] failed
> kafka.common.KafkaException: fetching topic metadata for topics
> [Set(test.app4)] from broker
> [ArrayBuffer(id:0,host:kafka.broker1,port:9092,
> id:1,host:kafka.broker2,port:9092)] failed
>     ...
> Caused by: java.net.ConnectException: Connection refused
>     ...
> 2014-04-02 13:31:12,419 DEBUG [kafka.producer.BrokerPartitionInfo] Getting
> broker partition info for topic test.app4
> 2014-04-02 13:31:12,419 DEBUG [kafka.producer.BrokerPartitionInfo]
> Partition [test.app4,0] has leader 1
> 2014-04-02 13:31:12,419 DEBUG [kafka.producer.async.DefaultEventHandler]
> Broker partitions registered for topic: test.app4 are 0
> 2014-04-02 13:31:12,419 DEBUG [kafka.producer.async.DefaultEventHandler]
> Sending 1 messages with no compression to [test.app4,0]
> 2014-04-02 13:31:12,420 DEBUG [kafka.producer.async.DefaultEventHandler]
> Producer sending messages with correlation id 238 for topics [test.app4,0]
> to broker 1 on kafka.broker2:9092
> 2014-04-02 13:31:12,423 ERROR [kafka.producer.SyncProducer] Producer
> connection to kafka.broker2:9092 unsuccessful
> java.net.ConnectException: Connection refused
>     ...
> WARN  [kafka.producer.async.DefaultEventHandler] Failed to send producer
> request with correlation id 238 to broker 1 with data for partitions
> [test.app4,0]
> java.net.ConnectException: Connection refused
>     ...
> INFO  [kafka.producer.async.DefaultEventHandler] Back off for 100 ms before
> retrying send. Remaining retries = 1
>
> 3. startup the kafka.broker2
> --> it fails in infinite loop too:
>
> INFO Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
> WARN [ReplicaFetcherThread-0-0], Error in fetch Name: FetchRequest;
> Version: 0; CorrelationId: 183; ClientId: ReplicaFetcherThread-0-0;
> ReplicaId: 1; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:
> [test.app4,0] -> PartitionFetchInfo(2,1048576)
> (kafka.server.ReplicaFetcherThread)
> java.net.ConnectException: Connection refused
>     ...
> [2014-04-02 12:56:33,816] INFO Reconnect due to socket error: null
> (kafka.consumer.SimpleConsumer)
>
> 5. shutdown kafka.broker1
> --> the initialization of kafka.broker2 is complete. System works fine.
>
> 6. startup kafka.broker1
> --> System works fine
>
>
> Please advise how to achieve high availability and what is wrong in this
> case.
>
> Regards,
> Anatoly
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message