kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: how to control assign policy for consumers
Date Thu, 29 May 2014 17:01:31 GMT
In the new consumer design the partitioning algorithm can be customized by
the user:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

Would that fit your use case?

Guozhang


On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:

> May be right place to add such optimization within
> https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ?
>
> -----Исходное сообщение-----
> От: Guozhang Wang [mailto:wangguoz@gmail.com]
> Отправлено: Wednesday, May 28, 2014 8:57 PM
> Кому: users@kafka.apache.org
> Тема: Re: how to control assign policy for consumers
>
> With high-level consumers you cannot control the assignment of partitions
> to consumers, and if partitions are located across all brokers each
> consumer is likely to connect to each of the brokers.
>
> One (little hacky) thing you can do is to manually assign the partitions
> to brokers such that each broker i of topic t is in broker i % 3, the for
> consumer fetching partition i of topic t, it will only connect to broker i
> % 3. By default the assignment is not aligned hence you will likely to have
> TCP between each pair of consumer/broker.
>
> Guozhang
>
>
> On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > How can I tell consumer to connect  to one broker  ...For example:
> >  -  I have 3 topic and 3 broker...also each topic have 6 partitions...
> >   - I than start 6 consumers to listen this 3 topics...And in such
> > case we can found from 6 (each consumer connect only to one broker) to
> > 18 (each consumer connect to every broker) TCP connection .... How I
> > can minimize network connections?
> >
> > With best regards, Aleksey Ryabkov
> >
> >
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message