kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryanne Dolan <ryannedo...@gmail.com>
Subject Re: Configuration guidelines for a specific use-case
Date Tue, 08 Jan 2019 17:52:06 GMT
Latency sounds high to me, maybe your JVMs are GC'ing a lot?

Ryanne

On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinovino@gmail.com
wrote:

> Hi expert,
>
>
> I would ask you some guidelines, web-pages or comments regarding my
> use-case.
>
>
> *Requirements*:
>
> - 2000+ producers
>
> - input rate 600k messages/s
>
> - consumers must write in 3 different databases (so i assume 3 consumer
> groups) at 600k messages/s overall (200k messages/s/database)
>
> - latency < 500ms between producers and databases
>
> - good availability
>
> - Possibility to process messages before to send them to the databases
> (Kafka stream? Of course in HA. Docker? Marathon?)
>
> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
> required), latency has higher priority
>
> - record size: 100-1000
>
>
> *Resources*:
>
> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
>
> producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
>
>
> *My* *configuration*:
>
> 3 brokers
>
> 6 partition (without replication in order to minimize latency)
>
> ack = 0 (missing data is tolerate)
>
> batch.size = 1024 (with 8196 the throughput is max)
>
> producers -> compression.type=none
>
>
>
> I did test using kafka-producer-perf-test.sh and
> kafka-consumer-perf-test.sh and i have a good throughput (500-600k
> messages/s using 3 producers and 3 consumers) but i would improve
> latency (0.3-2 sec) or features I'm not still considering.
>
>
> I thank you in advance.
>
> Cheers,
>
>
> Gioacchino
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message