samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Crim <rjc...@gmail.com>
Subject Re: ThreadJobFactory in production
Date Thu, 03 Mar 2016 04:17:11 GMT
Thanks for the clarification -- very helpful.

I'll take a look at those tickets!


On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan <nickpan47@gmail.com> wrote:

> Hi, Robert,
>
> The main reason that ThreadJobFactory and ProcessJobFactory are not
> considered "production-ready" is that there is only one container for the
> job and all tasks are assigned to the single container. Hence, it is not
> easy to scale out of a single host.
>
> As Rick mentioned, Netflix has put up a patch in SAMZA-41 based on 0.9.1 o
> allow static assignment of a subset of partitions to a single ProcessJob,
> which allows to launch multiple ProcessJobs in different hosts. We planned
> to merge it to 0.10. But it turns out that too much changes have gone into
> 0.10 and it became difficult to merge the patch. At this point, we can
> still try the following two options:
> 1) We can attempt to merge SAMZA-41 to 0.10.1 again, it may take some
> effort but would give a stop-gap solution.
> 2) We are working on a standalone Samza model (SAMZA-516, SAMZA-881) to
> allow users to run Samza w/o depending on YarnJobFactory. This is a
> long-term effort and will take some time to flesh out. Please join the
> discussion there s.t. we can be more aligned in our effort.
>
> Hope the above gives you an overall picture on where we are going.
>
> Thanks a lot!
>
> -Yi
>
> On Wed, Mar 2, 2016 at 1:28 PM, Rick Mangi <rick@chartbeat.com> wrote:
>
> > There was an interesting thread a while back from I believe the netflix
> > guys about running ThreadJobFactory in production.
> >
> >
> > > On Mar 2, 2016, at 4:20 PM, Robert Crim <rjcrim@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > We're currently working on a solution that allows us to run Samza jobs
> on
> > > Mesos. This seems to be going well, and something we'd like to move
> away
> > > from when native Mesos support is added to Samza.
> > >
> > > While we're developing and testing our scheduler, I'm wondering about
> the
> > > implications of running tasks with the ThreadJobFactory in
> "production".
> > > The documentation advise against this, but it's not clear why.
> > >
> > > If we were using the ThreadJobFactory inside of a docker container on
> > Mesos
> > > with Marathon for production, would be our main problem? These are not
> > > particularly high-load tasks. Aside from not be able to get
> find-grained
> > > resource scheduling per-task, it seems like the main issue the not
> being
> > to
> > > easily tell when a job stops due to error / exception.
> > >
> > > In other words, what would be stop-stopping reasons to not use the
> > > TreadJobFactory in production?
> > >
> > > Thanks,
> > > Rob
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message