kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Goodhope <kengoodh...@gmail.com>
Subject Re: Kafka/Hadoop consumers and producers
Date Fri, 09 Aug 2013 19:27:12 GMT
I just checked and that patch is in .8 branch.   Thanks for working on back
porting it Andrew.  We'd be happy to commit that work to master.

As for the kafka contrib project vs Camus, they are similar but not quite
identical.  Camus is intended to be a high throughput ETL for bulk
ingestion of Kafka data into HDFS.  Where as what we have in contrib is
more of a simple KafkaInputFormat.  Neither can really replace the other.
If you had a complex hadoop workflow and wanted to introduce some Kafka
data into that workflow, using Camus would be a gigantic overkill and a
pain to setup.  On the flipside, if what you want is frequent reliable
ingest of Kafka data into HDFS, a simple InputFormat doesn't provide you
with that.

I think it would be preferable to simplify the existing contrib
Input/OutputFormats by refactoring them to use the more stable higher level
Kafka APIs.  Currently they use the lower level APIs.  This should make
them easier to maintain, and user friendly enough to avoid the need for
extensive documentation.

Ken


On Fri, Aug 9, 2013 at 8:52 AM, Andrew Psaltis <psaltis.andrew@gmail.com>wrote:

> Dibyendu,
> According to the pull request: https://github.com/linkedin/camus/pull/15it was merged
into the camus-kafka-0.8
> branch. I have not checked if the code was subsequently removed, however,
> two at least one the important files from this patch (camus-api/src/main/java/com/linkedin/camus/etl/RecordWriterProvider.java)
> is still present.
>
> Thanks,
> Andrew
>
>
> On Fri, Aug 9, 2013 at 9:39 AM, <dibyendu.bhattacharya@pearson.com> wrote:
>
>> Hi Ken,
>>
>> I am also working on making the Camus fit for Non Avro message for our
>> requirement.
>>
>> I see you mentioned about this patch (
>> https://github.com/linkedin/camus/commit/87917a2aea46da9d21c8f67129f6463af52f7aa8)
>> which supports custom data writer for Camus. But this patch is not pulled
>> into camus-kafka-0.8 branch. Is there any plan for doing the same ?
>>
>> Regards,
>> Dibyendu
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Camus - Kafka ETL for Hadoop" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/camus_etl/KKS6t5-O-Ng/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> camus_etl+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Camus - Kafka ETL for Hadoop" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camus_etl+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message