kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karim Lamouri <ka...@lamouri.me>
Subject Unable to read messages after broker went down.
Date Fri, 01 Feb 2019 15:38:27 GMT
Hi,

One of our brokers went down (Kafka 1.1.1) and when it came back up some
consumers couldn’t read original topics using their the original consumer
group.

This doesn’t output anything:

bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
--group group_name

However, if I change the name of the group it does output data:

bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
--group group_name_renamed

I then tried to analyze the faulty group (making sure it’s not running at
the same time):

bin/kafka-consumer-groups --bootstrap-server server --describe --group
group_name
Error: Executing consumer group command failed due to The consumer
group command timed out while waiting for group to initialize

If I enable the logs for the kafka-console-consumer command here is the
endless logs that I get:

kafka-console-consumer --bootstrap-server ip1:9092,ip2:9092 --topic
ASSETS --group ASSETS_PROCESSOR_TOPOLOGY

[2018-12-26 15:47:15,243] INFO Registered
kafka:type=kafka.Log4jController MBean
(kafka.utils.Log4jControllerRegistration$)
[2018-12-26 15:47:15,387] INFO ConsumerConfig values:
    auto.commit.interval.ms = 5000
    auto.offset.reset = latest
    bootstrap.servers = [ip1:9092, ip2:9092]
    check.crcs = true
    client.id =
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = ASSETS_PROCESSOR_TOPOLOGY
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
 (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-12-26 15:47:15,579] INFO Kafka version : 2.0.1-cp1
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:15,579] INFO Kafka commitId : 3d167ab3fdad2e73
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:16,711] INFO Cluster ID: AkXF_gdZS5eVEEYbvUiyRg
(org.apache.kafka.clients.Metadata)
[2018-12-26 15:47:16,712] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address1:9092 (id: 2147483643 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:16,713] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Revoking previously assigned
partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-12-26 15:47:16,714] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:17,351] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address1:9092
(id: 2147483643 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,189] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,190] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,501] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,079] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,304] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,305] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,661] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,240] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,820] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,403] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,622] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,623] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,976] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,198] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,199] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,551] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[...]

Can someone help debug the issue?

Thank you
​

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message