kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niek Sanders <niek.sand...@gmail.com>
Subject Re: Kafka in AWS?
Date Wed, 21 Mar 2012 18:01:10 GMT
So what would you like the S3 files to actually look like?

One Kafka message body per line?  Should the message topic be tossed
in there too?

A tricky aspect is that the Kafka message body is an opaque byte
array.  For my own case I'm using JSON for the payload so it makes my
requirements simpler.

- Niek

On Tue, Mar 20, 2012 at 10:07 PM, Russell Jurney
<russell.jurney@gmail.com> wrote:
> I want events in S3 to process them in Hadoop. I'd like to emit them in my app, and have
them magically show up in 64MB chunks on S3. Like most everyone else.
> Russell Jurney http://datasyndrome.com

View raw message