kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: Deciding number of partitions
Date Sun, 11 Nov 2018 15:52:22 GMT
If I understood your question correctly - you are looking for recommended
guideline for #partitions / consumer.
Kafka doesn’t hold any responsibility for that - and I’m happy to be
corrected.

There are two partition assignment strategy - default and sticky. But you
should always aim for x/N partitions per consumer (in a given consumer
group).

x = number of consumers
N = number of partitions in a given topic.

This concept is irrelevant of Kafka. In any distributed ecosystem you
should maintain the above number such that

1) there’s no race between threads I.e. processing same task

2) tasks are evenly distributed amongst workers - unless a worker dies/
exits their pool.

So if you have not planned for an even distribution of partitions per
consumer thread, you may need to revisit how you have planned to spawn your
consumers in your cgroup.

I hope that makes sense.

Thanks,



On Sun, 11 Nov 2018 at 15:29, Sachit Murarka <connectsachit@gmail.com>
wrote:

> Hi,
>
> Thanks for reply.
> You have shared the command. I am looking for the formula or way to
> calculate the number of partitions for my topic.
>
> Regards,
> Sachit
>
> On Sun, 11 Nov 2018, 20:08 花儿向阳开 <2468946524@qq.com wrote:
>
> > Hi,
> >   Maybe you can use the command like:
> >     kafka-topic.sh --create --topic <topic_name> --partitions
> > <partition_number>  --replication-factor <replica_number> --zookeeper
> > <zookeeperIP:Port>
> >   I hope it helps!
> >
> >
> > Regards!
> > Cathy
> > ------------------ 原始邮件 ------------------
> > 发件人: "Sachit Murarka"<connectsachit@gmail.com>;
> > 发送时间: 2018年11月11日(星期天) 晚上10:48
> > 收件人: "users"<users@kafka.apache.org>;
> >
> > 主题: Deciding number of partitions
> >
> >
> >
> > Hello All,
> >
> > I am new to kafka.  Could anyone please tell how to decided number of
> > partitions for our topic?
> >
> >
> > Kind Regards,
> > Sachit Murarka
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message