kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shikhar Bhushan <shik...@confluent.io>
Subject Re: Kafka ETL for Parquet
Date Mon, 01 Aug 2016 22:40:38 GMT
Er, mislinked HDFS connector :)
https://github.com/confluentinc/kafka-connect-hdfs


On Mon, Aug 1, 2016 at 3:39 PM Shikhar Bhushan <shikhar@confluent.io> wrote:

> Hi Kidong,
>
> That's pretty cool! I'm curious what this offers over the Confluent HDFS
> connector <https://github.com/mykidong/kafka-etl-consumer>, though.
>
> The README mentions not depending on the Schema Registry, and that the
> schema can be retrieved via the classpath and Consul. This functionality
> should actually be pluggable with Connect by implementing a custom
> `Converter`, e.g. the SR comes with AvroConverter which acts as the glue.
> Converter classes can be specified with the `key.converter` and
> `value.converter` configs.
>
> Best,
>
> Shikhar
>
> On Mon, Aug 1, 2016 at 1:56 AM Kidong Lee <mykidong@gmail.com> wrote:
>
>> Hi,
>>
>> I have written a simple Kafka ETL which consumes avro encoded data from
>> Kafka and save them to Parquet on HDFS:
>> https://github.com/mykidong/kafka-etl-consumer
>>
>> It is implemented with Kafka Consumer API and Parquet Writer API.
>>
>> - Kidong Lee.
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message