kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "claude.warren@wipro.com" <claude.war...@wipro.com>
Subject Re: Determining broker max buffer size from producer.
Date Tue, 04 Sep 2018 08:16:44 GMT
Thank you for the reference but I don't see how it answers my question.


Basically, I am looking for a Java call that I can make from the producer to determine the
maximum message size set on the broker.


Assume a process is starting and it has the connection information for the broker so it can
create a producer.  While the producer can assume that the max buffersize is the same as the
default this may not be the case.  The buffer may have been increased or decreased.  My process
would like to know what the limit is so I can determine how to split very data streams being
serialized through kafka.  Is there a way to do this without reading the broker configuration
(which is probably not available to my process anyway)


Claude

________________________________
From: Satish Duggana <satish.duggana@gmail.com>
Sent: Monday, September 3, 2018 11:03:32 AM
To: users@kafka.apache.org
Subject: Re: Determining broker max buffer size from producer.

** This mail has been sent from an external source. Treat hyperlinks and attachments in this
email with caution**

Are you asking about the maximum message size that can be sent from a
producer?
max.message.bytes is used to control that at broker/topic level. You can
find more details
at the below links.

*"The largest record batch size allowed by Kafka. If this is increased and
there are consumers older than 0.10.2, the consumers' fetch size must also
be increased so that the they can fetch record batches this large. In the
latest message format version, records are always grouped into batches for
efficiency. In previous message format versions, uncompressed records are
not grouped into batches and this limit only applies to a single record in
that case."*


https://kafka.apache.org/documentation/#topicconfigs
https://kafka.apache.org/documentation/#brokerconfigs

Thanks,
Satish.

On Mon, Sep 3, 2018 at 2:12 PM, claude.warren@wipro.com <
claude.warren@wipro.com> wrote:

> Greetings,
>
>
> I have spent several hours looking through documentation and historical
> email chains and have been unable to find a solution to my problem.
>
>
> I have a case where I need to construct a producer and I need to know the
> broker max buffer size so that I don't get any nasty buffer too big errors
> from the server when serializing large objects.  I have been unable to find
> any way to do this short of reading the broker configuration and I can't do
> that from a remote call (as far as I can tell).  Does anybody have a
> solution?
>
>
> Claude
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com<http://www.wipro.com>
>
The information contained in this electronic message and any attachments to this message are
intended for the exclusive use of the addressee(s) and may contain proprietary, confidential
or privileged information. If you are not the intended recipient, you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately and destroy all copies
of this message and any attachments. WARNING: Computer viruses can be transmitted via email.
The recipient should check this email and any attachments for the presence of viruses. The
company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message