kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Kafka in AWS?
Date Tue, 20 Mar 2012 22:23:08 GMT
Yeah, that is the part I am hoping someone will contribute :)  I know I can
write that myself.  I also know it will be buggy and that I will have lots
of trouble.

If you contribute this code, it would be a huge boon to Kafka.  It is imo
the primary use case for Kafka atm... if only the code gets into git.

On Tue, Mar 20, 2012 at 3:04 PM, Niek Sanders <niek.sanders@gmail.com>wrote:

> Russell,
>
> I'm actually in the process of writing a Java code to go from Kafka
> messages to S3.  I might be able to rip-out my application-specific
> parts and share something later tonight.
>
> The biggest hassle is that you can't append to existing S3 files.  So
> unless you're planning on uploading each message as a separate S3
> object, this means you need message aggregation smarts on the Kafka
> consumer / S3 uploader side of things.
>
> Best,
> Niek
>
>
>
>
>
>
> On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
> <russell.jurney@gmail.com> wrote:
> > I wish someone would publish some source that writes events to S3.
> >
> > Russell Jurney
> > twitter.com/rjurney
> > russell.jurney@gmail.com
> > datasyndrome.com
> >
> > On Mar 20, 2012, at 11:20 AM, Dave Fayram <dfayram@gmail.com> wrote:
> >
> >> We've been successfully using Kafka on AWS as well, and JMX wise we
> >> just use an SSH tunnel.
> >>
> >> In general, we've been very happy with the performance on AWS, which
> >> some people have reservations about due to the I/O situation on most
> >> Amazon boxes.
> >>
> >> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
> >> <gautam.singaraju@gmail.com> wrote:
> >>> We are have been considering Kafka for a new Data Platform. Has someone
> >>> used Kafka in AWS? If so, could you please share your experiences with
> us?
> >>>
> >>> Thank you!
> >>> ---
> >>> Gautam
> >>
> >>
> >>
> >> --
> >> --
> >> Dave Fayram
> >> dfayram@gmail.com
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message