kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gioacchino Vino <gioacchinov...@gmail.com>
Subject Configuration guidelines for a specific use-case
Date Tue, 08 Jan 2019 17:45:25 GMT
Hi expert,


I would ask you some guidelines, web-pages or comments regarding my 
use-case.


*Requirements*:

- 2000+ producers

- input rate 600k messages/s

- consumers must write in 3 different databases (so i assume 3 consumer 
groups) at 600k messages/s overall (200k messages/s/database)

- latency < 500ms between producers and databases

- good availability

- Possibility to process messages before to send them to the databases 
(Kafka stream? Of course in HA. Docker? Marathon?)

- it's tolerate missing data ( 0.5% max ) (disk writing is not strictly 
required), latency has higher priority

- record size: 100-1000


*Resources*:

brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)

producers -> brokers -> consumers ( Bandwidth: 1 Gbps )


*My* *configuration*:

3 brokers

6 partition (without replication in order to minimize latency)

ack = 0 (missing data is tolerate)

batch.size = 1024 (with 8196 the throughput is max)

producers -> compression.type=none



I did test using kafka-producer-perf-test.sh and 
kafka-consumer-perf-test.sh and i have a good throughput (500-600k 
messages/s using 3 producers and 3 consumers) but i would improve 
latency (0.3-2 sec) or features I'm not still considering.


I thank you in advance.

Cheers,


Gioacchino


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message