kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alberto Miorin <amiorin78+ka...@gmail.com>
Subject Re: Alternative to camus
Date Fri, 13 Mar 2015 19:38:17 GMT
We are currently using spark streaming 1.2.1 with kafka and write-ahead log.
I will only say one thing : "a nightmare". ;-)

Let's see if things are better with 1.3.0 :
http://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html

On Fri, Mar 13, 2015 at 8:33 PM, William Briggs <wrbriggs@gmail.com> wrote:

> Spark Streaming also has built-in support for Kafka, and as of Spark 1.2,
> it supports using an HDFS write-ahead log to ensure zero data loss while
> streaming:
> https://databricks.com/blog/2015/01/15/improved-driver-fault-tolerance-and-zero-data-loss-in-spark-streaming.html
>
> -Will
>
> On Fri, Mar 13, 2015 at 3:28 PM, Alberto Miorin <amiorin78+kafka@gmail.com
> > wrote:
>
>> I'll try this too. It looks very promising.
>>
>> Thx
>>
>> On Fri, Mar 13, 2015 at 8:25 PM, Gwen Shapira <gshapira@cloudera.com>
>> wrote:
>>
>> > There's a KafkaRDD that can be used in Spark:
>> > https://github.com/tresata/spark-kafka. It doesn't exactly replace
>> > Camus, but should be useful in building Camus-like system in Spark.
>> >
>> > On Fri, Mar 13, 2015 at 12:15 PM, Alberto Miorin
>> > <amiorin78+kafka@gmail.com> wrote:
>> > > We use spark on mesos. I don't want to partition our cluster because
>> of
>> > one
>> > > YARN job (camus).
>> > >
>> > > Best
>> > >
>> > > Alberto
>> > >
>> > > On Fri, Mar 13, 2015 at 7:43 PM, Otis Gospodnetic <
>> > > otis.gospodnetic@gmail.com> wrote:
>> > >
>> > >> Just curious - why - is Camus not suitable/working?
>> > >>
>> > >> Thanks,
>> > >> Otis
>> > >> --
>> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log
>> Management
>> > >> Solr & Elasticsearch Support * http://sematext.com/
>> > >>
>> > >>
>> > >> On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin <
>> > amiorin78+kafka@gmail.com
>> > >> >
>> > >> wrote:
>> > >>
>> > >> > I was wondering if anybody has already tried to mirror a kafka
>> topic
>> > to
>> > >> > hdfs just copying the log files from the topic directory of the
>> broker
>> > >> > (like 00000000000023244237.log).
>> > >> >
>> > >> > The file format is very simple :
>> > >> > https://twitter.com/amiorin/status/576448691139121152/photo/1
>> > >> >
>> > >> > Implementing an InputFormat should not be so difficult.
>> > >> >
>> > >> > Any drawbacks?
>> > >> >
>> > >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message