kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: Deciding number of partitions
Date Sun, 11 Nov 2018 15:56:21 GMT
My mistake in explaining the formula

N / x partitions per consumer - the ideal ratio should be 1.


On Sun, 11 Nov 2018 at 15:52, M. Manna <manmedia@gmail.com> wrote:

> If I understood your question correctly - you are looking for recommended
> guideline for #partitions / consumer.
> Kafka doesn’t hold any responsibility for that - and I’m happy to be
> corrected.
>
> There are two partition assignment strategy - default and sticky. But you
> should always aim for x/N partitions per consumer (in a given consumer
> group).
>
> x = number of consumers
> N = number of partitions in a given topic.
>
> This concept is irrelevant of Kafka. In any distributed ecosystem you
> should maintain the above number such that
>
> 1) there’s no race between threads I.e. processing same task
>
> 2) tasks are evenly distributed amongst workers - unless a worker dies/
> exits their pool.
>
> So if you have not planned for an even distribution of partitions per
> consumer thread, you may need to revisit how you have planned to spawn your
> consumers in your cgroup.
>
> I hope that makes sense.
>
> Thanks,
>
>
>
> On Sun, 11 Nov 2018 at 15:29, Sachit Murarka <connectsachit@gmail.com>
> wrote:
>
>> Hi,
>>
>> Thanks for reply.
>> You have shared the command. I am looking for the formula or way to
>> calculate the number of partitions for my topic.
>>
>> Regards,
>> Sachit
>>
>> On Sun, 11 Nov 2018, 20:08 花儿向阳开 <2468946524@qq.com wrote:
>>
>> > Hi,
>> >   Maybe you can use the command like:
>> >     kafka-topic.sh --create --topic <topic_name> --partitions
>> > <partition_number>  --replication-factor <replica_number> --zookeeper
>> > <zookeeperIP:Port>
>> >   I hope it helps!
>> >
>> >
>> > Regards!
>> > Cathy
>> > ------------------ 原始邮件 ------------------
>> > 发件人: "Sachit Murarka"<connectsachit@gmail.com>;
>> > 发送时间: 2018年11月11日(星期天) 晚上10:48
>> > 收件人: "users"<users@kafka.apache.org>;
>> >
>> > 主题: Deciding number of partitions
>> >
>> >
>> >
>> > Hello All,
>> >
>> > I am new to kafka.  Could anyone please tell how to decided number of
>> > partitions for our topic?
>> >
>> >
>> > Kind Regards,
>> > Sachit Murarka
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message