kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetić <otis.gospodne...@gmail.com>
Subject Re: concept of record vs request vs batch
Date Tue, 14 Jun 2016 20:27:20 GMT
Hi,

Do you control the producers?  If so, couldn't you measure the message
sizes there?
Alternatively, you can use something like SPM for Kafka or other Kafka
monitoring tools that expose relevant metrics.

For example, I think you can compute avg msg size based on metrics shown in
these charts:
https://apps.sematext.com/spm-reports/s/T1aD50s1Xp
https://apps.sematext.com/spm-reports/s/x5WcTA470A

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Jun 13, 2016 at 4:43 PM, allen chan <allen.michael.chan@gmail.com>
wrote:

> In JMX for Kafka producer there are metrics for both request, record, and
> batch size Max + Avg.
>
> What is the difference between these concepts?
>
> In the logging use case: I assume record is the single log line, batch is
> multiple log lines together and request is the batch wrapped with the
> metadata to be sent.
>
> Can someone confirm or correct this assumption?
> I need to be able to see how big a message is so i can properly tune this
> setting: message.max.bytes. The confusion i have is do i use the request or
> the record metric.
>
>
> Thanks,
> --
> Allen Michael Chan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message