kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject HA: how to control assign policy for consumers
Date Thu, 29 May 2014 19:18:40 GMT

Well may be yes and may no....
I think that in design we must point out that co-ordinator also must analize connection graph
and must map consumers to broker with minimal network overhead...  

-----Исходное сообщение-----
От: Guozhang Wang [mailto:wangguoz@gmail.com] 
Отправлено: Thursday, May 29, 2014 9:02 PM
Кому: users@kafka.apache.org
Тема: Re: how to control assign policy for consumers

In the new consumer design the partitioning algorithm can be customized by the user:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

Would that fit your use case?

Guozhang


On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич < a.ryabkov@ntc-vulkan.ru>
wrote:

> May be right place to add such optimization within 
> https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Wednesday, May 28, 2014 8:57 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> With high-level consumers you cannot control the assignment of 
> partitions to consumers, and if partitions are located across all 
> brokers each consumer is likely to connect to each of the brokers.
>
> One (little hacky) thing you can do is to manually assign the 
> partitions to brokers such that each broker i of topic t is in broker 
> i % 3, the for consumer fetching partition i of topic t, it will only 
> connect to broker i % 3. By default the assignment is not aligned 
> hence you will likely to have TCP between each pair of consumer/broker.
>
> Guozhang
>
>
> On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <

> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > How can I tell consumer to connect  to one broker  ...For example:
> >  -  I have 3 topic and 3 broker...also each topic have 6 partitions...
> >   - I than start 6 consumers to listen this 3 topics...And in such 
> > case we can found from 6 (each consumer connect only to one broker) 
> > to
> > 18 (each consumer connect to every broker) TCP connection .... How I 
> > can minimize network connections?
> >
> > With best regards, Aleksey Ryabkov
> >
> >
>
>
> --
> -- Guozhang
>



--
-- Guozhang
Mime
View raw message