kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Булат Юсупов <biusu...@avtodoria.ru>
Subject Kafka cluster loses messages after zookeeper restart
Date Tue, 06 Mar 2018 09:00:01 GMT
Hi,

I'm starting a cluster of kafka brokers using Docker (5 brokers for
example, one broker per container). Kafka version 2.12-0.11.0.0, Zookeeper
3.4.10.

*The scenario:*

   - Starting 1st broker with config below

*zoo.cfg*

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/opt/zookeeper/data

clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888

*server.properties*

broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker1_IP:broker1_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0

*producer.properties*

bootstrap.servers=localhost:9092
compression.type=none

*consumer.properties*

zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group


   -

   Zookeeper is started in standalone mode, then starts kafka
   -

   Creating topic

/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic my-test-topic1

   - Sending message

echo "test_kafka1" | /opt/kafka/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic my-test-topic1

   - Checking message

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning --topic my-test-topic1 --max-messages 1

*Message is recieved*

   - Describe the topic

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181
--topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1
ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 1
Replicas: 1 Isr: 1

   - Starting rest 4 brokers

*zoo.cfg* on each broker from 1st to 5th (only 0.0.0.0:2888:3888 position
differs)

tickTime=2000
initLimit=10
syncLimit=5

dataDir=/opt/zookeeper/data

clientPort=2181
maxClientCnxns=10
minSessionTimeout=4000
maxSessionTimeout=1000000
server.1=0.0.0.0:2888:3888
server.2=broker2_IP:broker2_2888:broker2_3888
server.3=broker3_IP:broker3_2888:broker3_3888
server.4=broker4_IP:broker4_2888:broker4_3888
server.5=broker5_IP:broker5_2888:broker5_3888

*server.properties* on each broker from 1st to 5th (broker.id are unique,
broker_IP:broker_PORT differs for ech broker)

broker.id=N
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://broker_IP:broker_PORT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824log.retention.check.interval.ms=300000
zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.initial.rebalance.delay.ms=0

*producer.properties* on each broker from 1st to 5th

bootstrap.servers=localhost:9092
compression.type=none

*consumer.properties* on each broker from 1st to 5th

zookeeper.connect=127.0.0.1:2181zookeeper.session.timeout.ms=6000zookeeper.connection.timeout.ms=1000000group.id=test-consumer-group


   -

   Restarting zookeeper on each broker to take effect of zoo.cfg
   -

   Zookeepers gather into cluster
   -

   Topic moved to broker 5

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181
--topic my-test-topic1 Topic:my-test-topic1 PartitionCount:1
ReplicationFactor:1 Configs: Topic: my-test-topic1 Partition: 0 Leader: 5
Replicas: 5 Isr: 5

Is it normal behavior? Or should it stay on broker 1?

   - Checking message on each broker

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
--from-beginning --topic my-test-topic1 --max-messages 1

*Message is lost* (message is not lost when topic stays on broker 1, so it
is floating situation)

Could you help me with this problem.

TIA


-- 


*Булат Юсупов */ Инженер-программист / Отдел разработки

biusupov@avtodoria.ru / +7 917 889 99 32
skype: usbulat

ООО "Автодория"
+7 843 524 74 12
Казань, Технопарк в сфере высоких технологий "ИТ-парк",
Петербургская, 52,
офис 303
www.avtodoria.ru

*Инновации спасают жизни!*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message