kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Péter Nagykátai <st4r.f1...@gmail.com>
Subject Re: Kafka topic partition distributing evenly on disks
Date Fri, 07 Aug 2020 10:49:36 GMT
Hello everybody,

Thank you for the detailed answers. My issue is partly answered here:




*This rule also applies to disk-level, which means that when a set
ofpartitions assigned to a specific broker, each of the disks will get
thesame number of partitions without considering the load of disks at
thattime.*

 I admit, I didn't provide enough info either.

So my problem is that an existing topic got a huge surge of events for this
week. I knew that'll happen and I modified the partition count.
Unfortunately, it occurred to me a bit later, that I'll likely need some
extra disk space. So I added an extra disk to each broker. The thing I
didn't know, that Kafka won't evenly distribute the partitions on the disks.
So the question still remains:
 Is there any way to have Kafka evenly distribute data on its disks?
Also, what options do I have *after *I'm in the situation I described
above? (preferably without deleting the topic)

Thanks!

On Fri, Aug 7, 2020 at 12:00 PM Yingshuan Song <songyingshuan@gmail.com>
wrote:

> Hi Peter,
> Agreed with Manoj and Vinicius, i think those rules led to this result :
>
> 1)the partitions of a topic - N and replication number - R determine the
> real partition-replica count of this topic, which is N * R;
> 2)   kafka can distribute partitions evenly among brokers, but it is based
> on the broker count when the topic was created, this is important.
> If we create a topic (N - 4, R - 3) in a kafka cluster which contains 3
> kafka brokers, then 4 * 3 / 3 = 4 partitions will be assigned to each
> broker.
> But if a new broker was added into this cluster and another topic (N - 4, R
> - 3) need to be created, then 4 * 3 / 4 = 3 partitions will be assigned to
> each broker.
> Kafka will not assign all those partitions to the new added broker even
> though it is idle and i think this is a shortcoming of kafka.
> This rule also applies to disk-level, which means that when a set of
> partitions assigned to a specific broker, each of the disks will get the
> same number of partitions without considering the load of disks at that
> time.
> 3) when producer send records to topics, how to chose partiton : 3-1) if a
> record has a key, then the partition number calculate according to the key;
> 3-2) if  records have no keys, then those records will be sent to each
> partition in turns. So, if there are lots of records with the same key, and
> those records will be sent to the same partition, and may take up a lot of
> disk space.
>
>
> hope this helps
>
> Vinicius Scheidegger <vinicius.scheidegger@gmail.com> 于2020年8月7日周五
> 上午6:10写道:
>
> > Hi Peter,
> >
> > AFAIK, everything depends on:
> >
> > 1) How you have configured your topic
> >   a) number of partitions (here I understand you have 15 partitions)
> >   b) partition replication configuration (each partition necessarily has
> a
> > leader - primary responsible to hold the data - and for reads and writes)
> > you can configure the topic to have a number of replicas
> > 2) How you publish messages to the topic
> >   a) The publisher is responsible to choose the partition. This can be
> done
> > consciously (by setting the partition id while sending the message to the
> > topic) or unconsciously (by using the DefaultPartitioner or any other
> > partitioner scheme).
> >
> > All messages sent to a specific partition will be written first to the
> > leader (meaning that the disk configured for the partition leader will
> > receive the load) and then replicated to the replica (followers).
> > Kafka does not automatically distribute the data equally to the different
> > brokers - you need to think about your architecture having that in mind.
> >
> > I hope it helps
> >
> > On Thu, Aug 6, 2020 at 10:23 PM Péter Nagykátai <st4r.f1sch@gmail.com>
> > wrote:
> >
> > > I initially started with one data disk (mounted solely to hold Kafka
> > data)
> > > and recently added a new one.
> > >
> > > On Thu, Aug 6, 2020 at 10:13 PM <Manoj.Agrawal2@cognizant.com> wrote:
> > >
> > > > What do you mean older disk ?
> > > >
> > > > On 8/6/20, 12:05 PM, "Péter Nagykátai" <st4r.f1sch@gmail.com>
> wrote:
> > > >
> > > >     [External]
> > > >
> > > >
> > > >     Yeah, but it doesn't do that. My "older" disks have ~70
> partitions,
> > > the
> > > >     newer ones ~5 partitions. That's why I'm asking what went wrong.
> > > >
> > > >     On Thu, Aug 6, 2020 at 8:35 PM <Manoj.Agrawal2@cognizant.com>
> > wrote:
> > > >
> > > >     > Kafka  evenly distributed number of partition on each disk so
> in
> > > > your case
> > > >     > every disk should have 3/2 topic partitions .
> > > >     > It is producer job to evenly produce data by partition key  to
> > > topic
> > > >     > partition .
> > > >     > How it partition key , it is auto generated or producer sending
> > key
> > > > along
> > > >     > with message .
> > > >     >
> > > >     >
> > > >     > On 8/6/20, 7:29 AM, "Péter Nagykátai" <st4r.f1sch@gmail.com>
> > > wrote:
> > > >     >
> > > >     >     [External]
> > > >     >
> > > >     >
> > > >     >     Hello,
> > > >     >
> > > >     >     I have a Kafka cluster with 3 brokers (v2.3.0) and each
> > broker
> > > > has 2
> > > >     > disks
> > > >     >     attached. I added a new topic (heavyweight) and was
> surprised
> > > > that
> > > >     > even if
> > > >     >     the topic has 15 partitions, those weren't distributed
> evenly
> > > on
> > > > the
> > > >     > disks.
> > > >     >     Thus I got one disk that's almost empty and the other
> almost
> > > > filled
> > > >     > up. Is
> > > >     >     there any way to have Kafka evenly distribute data on its
> > > disks?
> > > >     >
> > > >     >     Thank you!
> > > >     >
> > > >     >
> > > >     > This e-mail and any files transmitted with it are for the sole
> > use
> > > > of the
> > > >     > intended recipient(s) and may contain confidential and
> privileged
> > > >     > information. If you are not the intended recipient(s), please
> > reply
> > > > to the
> > > >     > sender and destroy all copies of the original message. Any
> > > > unauthorized
> > > >     > review, use, disclosure, dissemination, forwarding, printing
or
> > > > copying of
> > > >     > this email, and/or any action taken in reliance on the contents
> > of
> > > > this
> > > >     > e-mail is strictly prohibited and may be unlawful. Where
> > permitted
> > > by
> > > >     > applicable law, this e-mail and other e-mail communications
> sent
> > to
> > > > and
> > > >     > from Cognizant e-mail addresses may be monitored.
> > > >     > This e-mail and any files transmitted with it are for the sole
> > use
> > > > of the
> > > >     > intended recipient(s) and may contain confidential and
> privileged
> > > >     > information. If you are not the intended recipient(s), please
> > reply
> > > > to the
> > > >     > sender and destroy all copies of the original message. Any
> > > > unauthorized
> > > >     > review, use, disclosure, dissemination, forwarding, printing
or
> > > > copying of
> > > >     > this email, and/or any action taken in reliance on the contents
> > of
> > > > this
> > > >     > e-mail is strictly prohibited and may be unlawful. Where
> > permitted
> > > by
> > > >     > applicable law, this e-mail and other e-mail communications
> sent
> > to
> > > > and
> > > >     > from Cognizant e-mail addresses may be monitored.
> > > >     >
> > > >
> > > >
> > > > This e-mail and any files transmitted with it are for the sole use of
> > the
> > > > intended recipient(s) and may contain confidential and privileged
> > > > information. If you are not the intended recipient(s), please reply
> to
> > > the
> > > > sender and destroy all copies of the original message. Any
> unauthorized
> > > > review, use, disclosure, dissemination, forwarding, printing or
> copying
> > > of
> > > > this email, and/or any action taken in reliance on the contents of
> this
> > > > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > > > applicable law, this e-mail and other e-mail communications sent to
> and
> > > > from Cognizant e-mail addresses may be monitored.
> > > > This e-mail and any files transmitted with it are for the sole use of
> > the
> > > > intended recipient(s) and may contain confidential and privileged
> > > > information. If you are not the intended recipient(s), please reply
> to
> > > the
> > > > sender and destroy all copies of the original message. Any
> unauthorized
> > > > review, use, disclosure, dissemination, forwarding, printing or
> copying
> > > of
> > > > this email, and/or any action taken in reliance on the contents of
> this
> > > > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > > > applicable law, this e-mail and other e-mail communications sent to
> and
> > > > from Cognizant e-mail addresses may be monitored.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message