kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Skip Montanaro <skip.montan...@gmail.com>
Subject subscribe to a group of topics, all partitions
Date Wed, 06 Dec 2017 16:39:13 GMT
I'm using the Python API in my clients (Conda kafka-python package, v
1.3.3). I have one client which is kind of a catch-all, subscribing to
several topics. There is just one instance, but I pass the group_id
arg to the constructor because I want to control commit points. As I
understand it, use of group_id means some partition assignment
strategy comes into play. I'm also subscribing to a list of topics
using the subscribe method. Setup looks like this:

    consumer = kafka.KafkaConsumer(bootstrap_servers="mybroker:9092",
    consumer.subscribe(topics=["topic1", "topic2", "topic3", ...])

Sometimes it seems to receive messages from all partitions (0, 1, 2).
At other times, it seems to start up only listening to just one
partition. How do I unambiguously tell it, "listen to all partitions
for each topic"? How can I confirm that in my client? When I try
logging consumer.assignment(), it's always the empty set.

I happen to know I have three partitions, so I think I could create
TopicPartition instances for each topic/partition pair. Still, I don't
think I should rely on that knowledge. Looking in kafka.partitioner, I
didn't see a "GreedyPartitioner" object, or anything else which might
obviously be used to grab everything.


Skip Montanaro

View raw message