kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasa Trifunovic <sasa.s.trifuno...@gmail.com>
Subject Second consumer within the same thread gets stuck during poll call
Date Tue, 10 Dec 2019 13:52:37 GMT
 I have two Kafka consumers, subscribed to different topics and belonging
to the same consumer group, having different consumer ids, running in the
same thread. They are executing poll sequentially but after the first is
done second seems to be stuck in poll. I tried using proper Kafka broker
and embedded one for testing and nothing changes.I tried associating them
with different consumer groups and that seems to be working but
unfortunately, that is not a viable solution for me.

I found this in "Kafka: The Definitive Guide":

"You can’t have multiple consumers that belong to the same group in one
thread and you can’t have multiple threads safely use the same consumer.
One consumer per thread is the rule."

That quote directs me towards some form of thread cooperation due to the
specific order of message processing I need to do.

Can someone provide an explanation of why is it necessary to run different
consumers belonging to the same consumer group, subscribed to different
topics in separate threads?

Test code with embedded kafka:

@Test
public void pollTest() {
    kafkaAdmin.createTopics(topics1);
    kafkaAdmin.createTopics(topics2);

    Consumer<String, String> consumer1 = createConsumer(topics1);
    Consumer<String, String> consumer2 = createConsumer(topics2);

    consumer1.poll(200);
    consumer2.poll(200);
}

Version of kafka - 2.3.0

Thank you.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message