kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gioacchino Vino <gioacchinov...@gmail.com>
Subject Re: Configuration guidelines for a specific use-case
Date Wed, 09 Jan 2019 16:37:49 GMT
Hi Ryanne,


I just forgot to insert the "linger.ms=0" configuration.

I got this result:


50000000 records sent, 706793.701055 records/sec (67.41 MB/sec), 7.29 ms 
avg latency, 1245.00 ms max latency, 0 ms 50th, 3 ms 95th, 197 ms 99th, 
913 ms 99.9th.


it's pretty good but I would like to improve it just a bit.

Do you think using 6 partitions in a 3 broker cluster is a good choice?


Gioacchino


On 08/01/2019 18:52, Ryanne Dolan wrote:
> Latency sounds high to me, maybe your JVMs are GC'ing a lot?
>
> Ryanne
>
> On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinovino@gmail.com
> wrote:
>
>> Hi expert,
>>
>>
>> I would ask you some guidelines, web-pages or comments regarding my
>> use-case.
>>
>>
>> *Requirements*:
>>
>> - 2000+ producers
>>
>> - input rate 600k messages/s
>>
>> - consumers must write in 3 different databases (so i assume 3 consumer
>> groups) at 600k messages/s overall (200k messages/s/database)
>>
>> - latency < 500ms between producers and databases
>>
>> - good availability
>>
>> - Possibility to process messages before to send them to the databases
>> (Kafka stream? Of course in HA. Docker? Marathon?)
>>
>> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
>> required), latency has higher priority
>>
>> - record size: 100-1000
>>
>>
>> *Resources*:
>>
>> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
>>
>> producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
>>
>>
>> *My* *configuration*:
>>
>> 3 brokers
>>
>> 6 partition (without replication in order to minimize latency)
>>
>> ack = 0 (missing data is tolerate)
>>
>> batch.size = 1024 (with 8196 the throughput is max)
>>
>> producers -> compression.type=none
>>
>>
>>
>> I did test using kafka-producer-perf-test.sh and
>> kafka-consumer-perf-test.sh and i have a good throughput (500-600k
>> messages/s using 3 producers and 3 consumers) but i would improve
>> latency (0.3-2 sec) or features I'm not still considering.
>>
>>
>> I thank you in advance.
>>
>> Cheers,
>>
>>
>> Gioacchino
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message