kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Bukowinski <pmb...@gmail.com>
Subject Re: Whether kafka broker will have impact with 2 MB message size
Date Thu, 14 Mar 2019 02:58:46 GMT
We have many production clusters with three topics in the 1-3MB range and the rest in the multi-kb
to sub-kb range. We do use gzip compression, implemented at the broker rather than the producer
level. The clusters don’t usually break a sweat. We use MirrorMaker to aggregate these topics
to a large central cluster and the message sizes aren’t an issue there, either.

The main problem we have with these large topics — especially one of them that is high throughput
— is that after a cluster has been up a while and partitions have moved around (we use cruise-control)
due to hardware failures, the topic tends to become less balanced across the broker storage
and some brokers/disks fill up faster than others.


> On Mar 13, 2019, at 6:28 PM, 1095193290@qq.com wrote:
> We have a use case where we want to produce data to kafka with max
> size of 2 MB rarely (That is, based on user operations message size
> will vary).
> Whether producing 2 Mb size will have any impact or we need to split
> the message to small chunk such as 100 KB and produce.
> If we produce into small chunk, it will increase response time for the
> user. Also, we have checked by producing 2 MB message to kafka and we
> doesn't see much latency there.
> Anyway if we split the data and produce, it doesn't have any impact in
> disk size. But whether broker performance will degrade due to this?
> Our broker configuration is:
> RAM 125.6 GB Disk Size 2.9 TB Processors 40
> Thanks,
> Hemnath K B.

View raw message