kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From svante karlsson <s...@csi.se>
Subject Isr difference between Metadata Response vs /kafka-topics.sh --describe
Date Wed, 21 Jan 2015 15:10:11 GMT
We are running an external (like in non supported) C++ client library
agains 0.8.2-rc2 and see differences in the Isr vector in Metadata Response
compared to what ./kafka-topics.sh --describe returns.

We have a triple replicated topic that is not updated during the test.

kafka-topics.sh
returns

        Topic: saka.test.int_datastream Partition: 0    Leader: 3
Replicas: 3,1,2 Isr: 2,1,3
        Topic: saka.test.int_datastream Partition: 1    Leader: 1
Replicas: 1,2,3 Isr: 2,1,3


After some debugging of the received packet it seems the data is actually
missing from the server.

After a sequensial restart of each broker - everything was back to normal

two pairs of loglines every 10s

initial state:

saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,

restart broker 1

handle_connect_retry_timer
_connect_async_next z8r102-mc12-4-4.sth-tc2.videoplaza.net:9092

saka.test.int_datastream Partition: 1 Leader: 2 Replicas: 1, 2, 3, Isr: 2,
3,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 2 Replicas: 1, 2, 3, Isr: 2,
3,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
...
saka.test.int_datastream Partition: 1 Leader: 2 Replicas: 1, 2, 3, Isr: 2,
3,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 2 Replicas: 1, 2, 3, Isr: 2,
3,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
3, 1,

saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
3,

restart broker 3

known brokers changed {....  }
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, Isr: 2, 1,
saka.test.int_datastream Partition: 0 Leader: 2 Replicas: 1, 2, Isr: 2, 1,
known brokers changed { .... }
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 2 Replicas: 3, 1, 2, Isr: 2,
1,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 2 Replicas: 3, 1, 2, Isr: 2,
1,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 2 Replicas: 3, 1, 2, Isr: 2,
1,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 2,
1,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 2,
1, 3,

restart broker 2

saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 1 Leader: 1 Replicas: 1, 2, 3, Isr: 1,
3, 2,
saka.test.int_datastream Partition: 0 Leader: 3 Replicas: 3, 1, 2, Isr: 1,
3, 2,


all this time kafka-topics.sh returns (except for a very short time during
the restart)

Topic: saka.test.int_datastream Partition: 0    Leader: 3       Replicas:
3,1,2 Isr: 2,1,3
Topic: saka.test.int_datastream Partition: 1    Leader: 1       Replicas:
1,2,3 Isr: 2,1,3


This seems reproducible by shutting down all brokers at the same time. Then
the isr vectors will never "heal". Bumping broker by broker heals them
again.

/svante


/svante

/svante

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message