kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: Dealing with large messages
Date Tue, 06 Oct 2015 15:45:11 GMT
Storing large blobs in S3 or HDFS and placing URIs in Kafka is the most
common solution I've seen in use.

On Tue, Oct 6, 2015 at 8:32 AM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> The best practice I think is to just put large objects in a blob store
> and have messages embed references to those blobs. Interestingly we
> ended up having to implement large-message-support at LinkedIn but for
> various reasons were forced to put messages inline (i.e., against the
> above recommendation). So we ended up having to break up large
> messages into smaller chunks. This obviously adds considerable
> complexity to the consumer since the checkpointing can become pretty
> complicated. There are other nuances as well - we can probably do a
> short talk on this at an upcoming meetup.
>
> Joel
>
>
> On Mon, Oct 5, 2015 at 9:31 PM, Rahul Jain <rahulj51@gmail.com> wrote:
> > In addition to the config changes mentioned in that post, you may also
> have
> > to change producer config if you are using the new producer.
> >
> > Specifically, *max.request.size* and *request.timeout.ms
> > <http://request.timeout.ms>* have to be increased to allow the producer
> to
> > send large messages.
> >
> >
> > On 6 Oct 2015 02:02, "James Cheng" <jcheng@tivo.com> wrote:
> >
> >> Here’s an article that Gwen wrote earlier this year on handling large
> >> messages in Kafka.
> >>
> >> http://ingest.tips/2015/01/21/handling-large-messages-kafka/
> >>
> >> -James
> >>
> >> > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota <pradeepg26@gmail.com>
> >> wrote:
> >> >
> >> > Fellow Kafkaers,
> >> >
> >> > We have a pretty heavyweight legacy event logging system for batch
> >> > processing. We're now sending the events into Kafka now for realtime
> >> > analytics. But we have some pretty large messages (> 40 MB).
> >> >
> >> > I'm wondering if any of you have use cases where you have to send
> large
> >> > messages to Kafka and how you're dealing with them.
> >> >
> >> > Thanks,
> >> > Pradeep
> >>
> >>
> >> ________________________________
> >>
> >> This email and any attachments may contain confidential and privileged
> >> material for the sole use of the intended recipient. Any review,
> copying,
> >> or distribution of this email (or any attachments) by others is
> prohibited.
> >> If you are not the intended recipient, please contact the sender
> >> immediately and permanently delete this email and any attachments. No
> >> employee or agent of TiVo Inc. is authorized to conclude any binding
> >> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> >> Inc. may only be made by a signed written agreement.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message