kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karim Lamouri <ka...@lamouri.me>
Subject Re: Cannot read messages using a specific group
Date Wed, 26 Dec 2018 14:58:43 GMT
Hi all,

I set the log level to INFO and here are the logs.

I’m not sure why those logs say that the Group coordinator is unavailable
or invalid because the brokers are all up (including the ones with the ip
specified in the log) and just changing the consumer group makes it work.
If anybody had an idea, that would be very helpful. (also committing the
offsets to Zookeeper works, so I'm wondering if there is a bug with
__Consummer_Offset)

Thank you for your help

kafka-console-consumer --bootstrap-server ip1:9092,ip2:9092 --topic
ASSETS --group ASSETS_PROCESSOR_TOPOLOGY

[2018-12-26 15:47:15,243] INFO Registered
kafka:type=kafka.Log4jController MBean
(kafka.utils.Log4jControllerRegistration$)
[2018-12-26 15:47:15,387] INFO ConsumerConfig values:
    auto.commit.interval.ms = 5000
    auto.offset.reset = latest
    bootstrap.servers = [ip1:9092, ip2:9092]
    check.crcs = true
    client.id =
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = ASSETS_PROCESSOR_TOPOLOGY
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
 (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-12-26 15:47:15,579] INFO Kafka version : 2.0.1-cp1
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:15,579] INFO Kafka commitId : 3d167ab3fdad2e73
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:16,711] INFO Cluster ID: AkXF_gdZS5eVEEYbvUiyRg
(org.apache.kafka.clients.Metadata)
[2018-12-26 15:47:16,712] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address1:9092 (id: 2147483643 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:16,713] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Revoking previously assigned
partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-12-26 15:47:16,714] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:17,351] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address1:9092
(id: 2147483643 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,189] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,190] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,501] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,079] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,304] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,305] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,661] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,240] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,820] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,403] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,622] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,623] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,976] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,198] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,199] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,551] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)


On Thu, Dec 20, 2018 at 1:23 PM Karim Lamouri <karim@lamouri.me> wrote:

> Hi Ryanne,
>
> Here is the output of the command:
>
> bin/kafka-consumer-groups --bootstrap-server server --describe --group group_name
> Error: Executing consumer group command failed due to The consumer group command timed
out while waiting for group to initialize
>
> The group_name consumer group is not running in the cluster. What is
> noteworthy is that the consumer group group_name cannot be reset by the
> CLI (to either latest or earliest) and is still not working way past the
> retention period of all the topics (from the consumers with latest or
> earliest auto.offset.reset).
>
> Here is the output:
>
> bin/kafka-consumer-groups --bootstrap-server server --group group_name --reset-offsets
--to-latest --execute
> Error: Executing consumer group command failed due to The consumer group command timed
out while waiting for group to initialize
>
> ​
> Karim
>
> On Tue, Dec 18, 2018 at 8:27 PM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
>
>> Karim, can you inspect the offsets using kafka-consumer-groups.sh and let
>> us know what you see?
>>
>> Also, is it possible that "group_name" is being used by a consumer group
>> that is still running? For example, say you have a topic with one
>> partition
>> and a consumer with "group_name" already. In that case, a second console
>> consumer with "group_name" wouldn't find any partitions to consume.
>>
>> Ryanne
>>
>> On Tue, Dec 18, 2018 at 11:18 AM Karim Lamouri <karim@lamouri.me> wrote:
>>
>> > Hi,
>> >
>> > One of our brokers went down and when it came back up the consumer of
>> one
>> > topic couldn’t read using the original consumer group.
>> >
>> > This doesn’t output anything:
>> >
>> > bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
>> > --group group_name
>> >
>> > However, if I change the name of the group it does output data:
>> >
>> > bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
>> > --group group_name_renamed
>> >
>> > Can someone help debug the issue? We are running Kafka 1.1.1
>> >
>> > Thank you
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message