kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject HA: how to control assign policy for consumers
Date Thu, 29 May 2014 15:59:02 GMT
May be right place to add such optimization within https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics
?

-----Исходное сообщение-----
От: Guozhang Wang [mailto:wangguoz@gmail.com] 
Отправлено: Wednesday, May 28, 2014 8:57 PM
Кому: users@kafka.apache.org
Тема: Re: how to control assign policy for consumers

With high-level consumers you cannot control the assignment of partitions to consumers, and
if partitions are located across all brokers each consumer is likely to connect to each of
the brokers.

One (little hacky) thing you can do is to manually assign the partitions to brokers such that
each broker i of topic t is in broker i % 3, the for consumer fetching partition i of topic
t, it will only connect to broker i % 3. By default the assignment is not aligned hence you
will likely to have TCP between each pair of consumer/broker.

Guozhang


On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru>
wrote:

> How can I tell consumer to connect  to one broker  ...For example:
>  -  I have 3 topic and 3 broker...also each topic have 6 partitions...
>   - I than start 6 consumers to listen this 3 topics...And in such 
> case we can found from 6 (each consumer connect only to one broker) to 
> 18 (each consumer connect to every broker) TCP connection .... How I 
> can minimize network connections?
>
> With best regards, Aleksey Ryabkov
>
>


--
-- Guozhang
Mime
View raw message