kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From girija arumugam <girijaarumuga...@gmail.com>
Subject Regarding Fairness in choosing topic and partition for an event
Date Tue, 01 Dec 2020 10:52:04 GMT
Team,
 We are having a use-case, where a User in an Organisation can send data at
different rates for the event - sending mail.These events are published to
kafka and are consumed at our application side for processing.We need to
maintain the order for the events produced by a user in an org.

In our case, the growing factors are

   1. Organisation - 1 to million.
   2. Each Org can have Users from 1 to million.

*Topic Design :*

   1. .Fixed number of topics with fixed number of partitions : - 10 topics
   with 10 partitions
   2. 2.Single topic with N partitions

In these designs, we are not able to provide fairness to all the
events/messages.

(ie) Take an example,

     Let's say there are two orgs o1, o2 . Each org has 2 users. u1,u2
belong to o1 and u3,u4 belong to o2.And I have a single topic with 5
partitions.

    Let's assume that my partition logic for the example , the produce rate
and its state

   - o1,u1 -> 0th partition -> produce data at the rate of 1000
   messages/sec -> hyperactive
   - o1,u2 -> 1st partition -> produce data at the rate of 10 messages/sec
   -> normal
   - o2,u3 -> 0th partition -> produce data at the rate of 5 messages/min
   -> normal
   - o2,u4 -> 2nd partition -> produce data at the rate of 50 messages/sec
   -> normal

 Here, o1,u1 and o2,u3 are producing data to the same partition at
different rates.In this case, I want to give fairness to o1,u1 &
o2,u3.o1,u1 will be the one who is using the 0th partition more than
o2,u3.So , if i get a message from o2,u3 , I need to give some high
priority to the message.

I would like to design a topic & partition in such a way that every message
produced should be equally treated and the message from one org-one user
has to be maintained in an order.

*Questions ?*

   1. How to design a topic for an event to give fairness in-terms of  orgs
   as well as users ?
   2. We know that at the end, the event will be residing on one of the
   partitions in the topic.How to choose a partition for an event created by a
   user in an org ? How to give fairness in choosing a partition for a user ?
   3. *How to achieve one/more hyperactive user(s) from an org doesn't
   affect the others ?*


Anyone, please guide me to achieve a design.

I have referred the following references,

   1.
   https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/
   2. https://www.confluent.io/blog/put-several-event-types-kafka-topic/


Regards,
Girija A.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message