kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niek Sanders <niek.sand...@gmail.com>
Subject Re: Kafka in AWS?
Date Tue, 20 Mar 2012 22:04:04 GMT
Russell,

I'm actually in the process of writing a Java code to go from Kafka
messages to S3.  I might be able to rip-out my application-specific
parts and share something later tonight.

The biggest hassle is that you can't append to existing S3 files.  So
unless you're planning on uploading each message as a separate S3
object, this means you need message aggregation smarts on the Kafka
consumer / S3 uploader side of things.

Best,
Niek






On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
<russell.jurney@gmail.com> wrote:
> I wish someone would publish some source that writes events to S3.
>
> Russell Jurney
> twitter.com/rjurney
> russell.jurney@gmail.com
> datasyndrome.com
>
> On Mar 20, 2012, at 11:20 AM, Dave Fayram <dfayram@gmail.com> wrote:
>
>> We've been successfully using Kafka on AWS as well, and JMX wise we
>> just use an SSH tunnel.
>>
>> In general, we've been very happy with the performance on AWS, which
>> some people have reservations about due to the I/O situation on most
>> Amazon boxes.
>>
>> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
>> <gautam.singaraju@gmail.com> wrote:
>>> We are have been considering Kafka for a new Data Platform. Has someone
>>> used Kafka in AWS? If so, could you please share your experiences with us?
>>>
>>> Thank you!
>>> ---
>>> Gautam
>>
>>
>>
>> --
>> --
>> Dave Fayram
>> dfayram@gmail.com

Mime
View raw message