kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Jain <rahul...@gmail.com>
Subject Re: Dealing with large messages
Date Tue, 06 Oct 2015 04:31:35 GMT
In addition to the config changes mentioned in that post, you may also have
to change producer config if you are using the new producer.

Specifically, *max.request.size* and *request.timeout.ms
<http://request.timeout.ms>* have to be increased to allow the producer to
send large messages.


On 6 Oct 2015 02:02, "James Cheng" <jcheng@tivo.com> wrote:

> Here’s an article that Gwen wrote earlier this year on handling large
> messages in Kafka.
>
> http://ingest.tips/2015/01/21/handling-large-messages-kafka/
>
> -James
>
> > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota <pradeepg26@gmail.com>
> wrote:
> >
> > Fellow Kafkaers,
> >
> > We have a pretty heavyweight legacy event logging system for batch
> > processing. We're now sending the events into Kafka now for realtime
> > analytics. But we have some pretty large messages (> 40 MB).
> >
> > I'm wondering if any of you have use cases where you have to send large
> > messages to Kafka and how you're dealing with them.
> >
> > Thanks,
> > Pradeep
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message