kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Keylis <vkeylis2...@gmail.com>
Subject Re: Kafka/Hadoop consumers and producers
Date Wed, 03 Jul 2013 08:20:25 GMT
What is the difference between this project and Camus? Which advantages to use for loading
log entries from kafka into Hadoop ?


Sent from my iPhone

On Jul 2, 2013, at 5:01 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> We currently have a contrib package for consuming and producing messages
> from mapreduce (
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tree;f=contrib;h=e53e1fb34893e733b10ff27e79e6a1dcbb8d7ab0;hb=HEAD
> ).
> We keep running into problems (e.g. KAFKA-946) that are basically due to
> the fact that the Kafka committers don't seem to mostly be Hadoop
> developers and aren't doing a good job of maintaining this code (keeping it
> tested, improving it, documenting it, writing tutorials, getting it moved
> over to the more modern apis, getting it working with newer Hadoop
> versions, etc).
> A couple of options:
> 1. We could try to get someone in the Kafka community (either a current
> committer or not) who would adopt this as their baby (it's not much code).
> 2. We could just let Camus take over this functionality. They already have
> a more sophisticated consumer and the producer is pretty minimal.
> So are there any people who would like to adopt the current Hadoop contrib
> code?
> Conversely would it be possible to provide the same or similar
> functionality in Camus and just delete these?
> -Jay

View raw message