kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Gollakota <pradeep...@gmail.com>
Subject Re: Kafka ETL Camus Question
Date Tue, 03 Feb 2015 07:15:18 GMT
Hi Bhavesh,

At Lithium, we don't run Camus in our pipelines yet, though we plan to. But
I just wanted to comment regarding speculative execution. We have it
disabled at the cluster level and typically don't need it for most of our
jobs. Especially with something like Camus, I don't see any need to run
parallel copies of the same task.

On Mon, Feb 2, 2015 at 10:36 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com>
wrote:

> Hi Jun,
>
> Thanks for info.  I did not get answer  to my question there so I thought I
> try my luck here :)
>
> Thanks,
>
> Bhavesh
>
> On Mon, Feb 2, 2015 at 9:46 PM, Jun Rao <jun@confluent.io> wrote:
>
> > You can probably ask the Camus mailing list.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Jan 29, 2015 at 1:59 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com>
> > wrote:
> >
> > > Hi Kafka Team or Linked-In  Team,
> > >
> > > I would like to know if you guys run Camus ETL job with speculative
> > > execution true or false.  Does it make sense to set this to false ?
> > Having
> > > true, it creates additional load on brokers for each map task (create a
> > map
> > > task to pull same partition twice).  Is there any advantage to this
> > having
> > > it on vs off ?
> > >
> > > mapred.map.tasks.speculative.execution
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message