kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chicolo, Robert (rchicolo@student.cccs.edu)" <rchic...@student.cccs.edu>
Subject Re: Choose the number of partitions/topics
Date Mon, 29 Jan 2018 20:19:00 GMT
so it goes beyond the throughput that kafka can support.  You have to decide as to what degree
of parallelism your application can support. If one message processing depends on processing
for another message, that limits the degree to which you can process in parallel. Depending
on how much time the processing of the message takes and the desired response times the stream
can be parallelized.

________________________________
From: Maria Pilar <pilifer78@gmail.com>
Sent: Monday, January 29, 2018 8:58:17 AM
To: dev@kafka.apache.org; users@kafka.apache.org
Subject: Choose the number of partitions/topics

Hi everyone

I have design an integration between 2 systems throug our API Stream Kafka,
and the requirements are unclear to choose properly the number of
partitions/topics.

That is the use case:

My producer will send 28 different type of events, so I have decided to
create 28 topics.

The max size value for one message will be 4,096 bytes and the total size
(MB/day) will be 2.469,888 mb/day.

The retention will be 2 days.

By default I´m thinking in one partition that as recomentation by confluent
it can produce 10 Mb/second.

However the requirement for the consumer is the minimun latency (sub 3
seconds), so I thinking to create more leader partitions/per topic to
paralle and achive the thoughput.

Do you know what is the best practice or formule to define it properly?

Thanks

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message