kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dimitry Lvovsky <dlvov...@gmail.com>
Subject Re: Dynamic Partitioning
Date Wed, 09 Jan 2019 10:58:00 GMT
One approach you can take is to set an upper bound number of partitions a
priori. Imagine your key was the username.  If you had 2 partitions in your
topic and 4 users writing messages, then Kafka would split these messages
between two partitions of the topic (assuming the usernames are unique).
For your design I would ask the questions:

   - how many simultaneous users will I have
   - how long does it take to process a message
   - how many messages per second will I have

Answering the above questions will allow you to define the expectations and
limits of your applications.
So you if know, for example,  that users will send  2 messages per second,
and say it takes 250ms to process a message, a user could wait potentially
wait for 1 ( assuming the messages are equally distributed -- which they
might not be)  second to receive a message. That might be fine or you might
want that to go faster.  If it's the latter, then build it from the
beginning with more partitions in mind.  There is nothing stopping you from
say declaring a topic with 64  partitions from day one  (or more, just
don't go crazy as to-many are not good either), but use only 2 consumers
from the start.  In this case, each consumer is reading from 32
partitions.  Now let's say more users sign on, and you have some kind of
KPI you want to meet, e.g. on average users shouldn't wait more than 500ms
to receive a message.  In this case, you can simply spin up more
consumers.  In the example above, say 8 users are now online, so you spin
up 2 more consumers -- now each consumer will have 16 partitions each.  You
can scale up or down by the number of consumers you have.

There can come a point when the demand on your system will be greater than
the original limits you set when doing your initial design.  This is the
point I would consider adding more partitions.

Hope this helps.

Dimitry

On Tue, Jan 8, 2019 at 6:29 PM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> Just key your messages by user ID and they'll be processed in sequence.
> Every message with the same key will go to the same partition, regardless
> of how many partitions there are.
>
> Ryanne
>
> On Tue, Jan 8, 2019, 8:57 AM marimuthu eee <marimuthueee76@gmail.com
> wrote:
>
> > Hi Ryanne Dolan,
> >
> > My requirement is to process notification messages in parallel.So i have
> > an idea to pick kafka as a messaging system.But i need dynamic
> partitioning*
> > to handle individual user notification messages in serial manner*.Is it
> > possible to create dynamic partition creation for a particular topic.
> >
> >
> >
> > On Tue, Jan 8, 2019 at 8:20 PM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
> >
> >> Can you explain why you think you need dynamic partitioning? Maybe we
> can
> >> help you design without it.
> >>
> >> Ryanne
> >>
> >> On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthueee76@gmail.com
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > My requirement is to process notification messages in parallel.So i
> >> have an
> >> > idea to pick kafka as a messaging system.But i need dynamic
> >> partitioning to
> >> > handle individual user notification messages.Is it possible to create
> >> > dynamic partition creation for a particular topic.
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message