kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix GV <fe...@mate1inc.com>
Subject Re: Kafka+Avro+Hadoop
Date Tue, 14 Feb 2012 22:29:45 GMT
Indeed, there has been no mention that the LinkedIn Kafka/Hadoop ETL code
has been released.

I'm glad to see that the little script I made is useful for others :) ...

If you want to consume Binary Avro and write it straight into Hadoop, you
should be able to use the regular hadoop-consumer contrib (or the
incremental-consumer, which is just a wrapper for the hadoop-consumer
contrib), no?

--
Felix



On Mon, Feb 13, 2012 at 4:18 AM, Sebastian Eichner <
mail@sebastian-eichner.de> wrote:

> Hello!
>
> We want to use Kafka to transport Avro messages, store them in
> Avro-Format on HDFS and process then with Hadoop.
>
> So far i have the Hadoop-Incremental-Consumer from Felix and a simple
> Avro-Producer working. My next step would be to make the
> Hadoop-Consumer receive Binary Avro and store it in files, but before
> diving into this i wanted to ask:
>
> Is there any other code for this scenario already available? So far i
> could not find anything from the list archives and google. In the
> archives i read that LinkedIn does something similiar but afaik it's
> not yet released.
>
> Thanks!
>  Sebastian
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message