kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Safaric <dominiksafa...@gmail.com>
Subject Re: Oversized Message 40k
Date Tue, 22 Nov 2016 12:18:14 GMT
Big is a relative term.

And the question you ask is quite difficult to answer because not other information is available
- including the configuration of the Kafka cluster, hardware specification etcetera. 

I suggest the following: (1) read a couple of benchmarks such as [1], (2) investigate onto
this question of matter yourself by designing and running a microbenchmark - since you’re
the only one who knows what throughput or for example latency you’d like to achieve.

But in general, I do believe that those messages are larger then the expected. Hence, you
may also want to take a look at compression in Kafka - although you’ll lose performance.

[1] https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

> On 22 Nov 2016, at 13:00, Felipe Santos <felipegs@gmail.com> wrote:
> I read on documentation that kafka is not optimized for big messages, what
> is considered a big message?
> For us the messages will be on average from 20k ~ 40k? Is this a real
> problem?
> Thanks
> -- 
> Felipe Santos

View raw message