kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Kafka in AWS?
Date Wed, 21 Mar 2012 05:07:27 GMT
I want events in S3 to process them in Hadoop. I'd like to emit them in my app, and have them
magically show up in 64MB chunks on S3. Like most everyone else.

Russell Jurney http://datasyndrome.com

On Mar 20, 2012, at 10:03 PM, Neha Narkhede <neha.narkhede@gmail.com> wrote:

> Russell,
> 
> By "sink events into S3", do you mean you want to have some plugin that
> will suck data out of your Kafka brokers and upload to S3. Would you mind
> describing use cases that would require to send data to Kafka, then upload
> data to S3, and then use it by querying S3 ?
> 
> Thanks,
> Neha
> On Mar 20, 2012 4:51 PM, "Russell Jurney" <russell.jurney@gmail.com> wrote:
> 
>> I think as soon as someone commits code that reliably sinks events to S3,
>> Kafka adoption will skyrocket.  There is no good solution to this yet.
>> MANY people want one.
>> 
>> Russ
>> 
>> On Tue, Mar 20, 2012 at 3:32 PM, Felix GV <felix@mate1inc.com> wrote:
>> 
>>> The primary use case for Kafka is to use it on AWS...???
>>> 
>>> Sorry if I put words you didn't intend in your mouth :P ... I just
>> thought
>>> that sounded funny ;)
>>> 
>>> Sorry for being off-topic. Carry on :/ !
>>> 
>>> --
>>> Felix
>>> 
>>> 
>>> 
>>> On Tue, Mar 20, 2012 at 6:23 PM, Russell Jurney <
>> russell.jurney@gmail.com
>>>> wrote:
>>> 
>>>> Yeah, that is the part I am hoping someone will contribute :)  I know I
>>> can
>>>> write that myself.  I also know it will be buggy and that I will have
>>> lots
>>>> of trouble.
>>>> 
>>>> If you contribute this code, it would be a huge boon to Kafka.  It is
>> imo
>>>> the primary use case for Kafka atm... if only the code gets into git.
>>>> 
>>>> On Tue, Mar 20, 2012 at 3:04 PM, Niek Sanders <niek.sanders@gmail.com
>>>>> wrote:
>>>> 
>>>>> Russell,
>>>>> 
>>>>> I'm actually in the process of writing a Java code to go from Kafka
>>>>> messages to S3.  I might be able to rip-out my application-specific
>>>>> parts and share something later tonight.
>>>>> 
>>>>> The biggest hassle is that you can't append to existing S3 files.  So
>>>>> unless you're planning on uploading each message as a separate S3
>>>>> object, this means you need message aggregation smarts on the Kafka
>>>>> consumer / S3 uploader side of things.
>>>>> 
>>>>> Best,
>>>>> Niek
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
>>>>> <russell.jurney@gmail.com> wrote:
>>>>>> I wish someone would publish some source that writes events to S3.
>>>>>> 
>>>>>> Russell Jurney
>>>>>> twitter.com/rjurney
>>>>>> russell.jurney@gmail.com
>>>>>> datasyndrome.com
>>>>>> 
>>>>>> On Mar 20, 2012, at 11:20 AM, Dave Fayram <dfayram@gmail.com>
>> wrote:
>>>>>> 
>>>>>>> We've been successfully using Kafka on AWS as well, and JMX wise
>> we
>>>>>>> just use an SSH tunnel.
>>>>>>> 
>>>>>>> In general, we've been very happy with the performance on AWS,
>> which
>>>>>>> some people have reservations about due to the I/O situation
on
>> most
>>>>>>> Amazon boxes.
>>>>>>> 
>>>>>>> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
>>>>>>> <gautam.singaraju@gmail.com> wrote:
>>>>>>>> We are have been considering Kafka for a new Data Platform.
Has
>>>> someone
>>>>>>>> used Kafka in AWS? If so, could you please share your experiences
>>>> with
>>>>> us?
>>>>>>>> 
>>>>>>>> Thank you!
>>>>>>>> ---
>>>>>>>> Gautam
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> --
>>>>>>> Dave Fayram
>>>>>>> dfayram@gmail.com
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>> datasyndrome.com
>> 

Mime
View raw message