kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dana Powers <dana.pow...@gmail.com>
Subject Re: unable to set consumer group
Date Wed, 20 Jan 2016 16:57:58 GMT
version 0.9.5 of kafka-python does not support coordinated consumer groups.
You can get this feature in the master branch on github (
https://github.com/dpkp/kafka-python) using kafka 0.9.0.0 brokers. I expect
to release the updates to pypi soon, but for now you'll have to install
from source.

Other python alternatives: assign partitions statically to each instance
via chef/ansible/etc; or try pykafka's BalancedConsumer implementation,
which uses Zookeeper to coordinate assignment.

-Dana

On Wed, Jan 20, 2016 at 2:35 AM, Ilja Golshtein <ilejn@yandex.ru> wrote:

> Hello.
>
> I am trying to create consumer using kafka_python-0.9.5.
>
> I expect that several instances of the script
>
> ==
> consumer = KafkaConsumer('some-topic',
>                          auto_offset_reset='largest',
>                          metadata_broker_list=['localhost:9092'],
>                          group_id='vasya_group',
>                          auto_commit_enable=True)
> consumer.set_topic_partitions('some-topic')
> while True:
>     msg=consumer.next()
>     print(msg)
> ==
> would receive different messages, while in reality every instance receives
> all messages.
>
> What can be done to achieve
> "Consumers label themselves with a consumer group name, and each message
> published to a topic is delivered to one consumer instance within each
> subscribing consumer group" as advertised in Kafka Documentation?
>
> I am 100% that messages are distributed among several partitions (namely
> 10).
>
> Thanks.
>
> --
> Best regards
> Ilja Golshtein
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message