hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Suresh <asur...@apache.org>
Subject Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider
Date Tue, 19 Jul 2016 03:14:01 GMT
+1
It is definitely a step in the right direction and it's about time YARN
recognizes services as first-class citizens.
I would also like to see this effort encompass not just services but also
general long-lived applications such as say storm/heron topologies, samza
and possibly support for frameworks like REEF.



On Mon, Jul 18, 2016 at 6:32 PM, Sangjin Lee <sjlee@apache.org> wrote:

> +1. It would be great if it becomes the default way for services to run on
> YARN. That would also make it easier to make YARN features support services
> better. As you mentioned, there would be a lot of logistics to work out,
> and we'll need to talk about them in more detail as they come up.
>
> Regards,
> Sangjin
>
> On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gsaha@hortonworks.com> wrote:
>
> > +1 forwarding from the discussion in Slider DL
> >
> > Note: On the Slider side, we plan to create a branch corresponding to
> this
> > YARN branch. In this, we intend to remove all the pieces which will move
> > to the YARN branch (Slider core, AM, client). We will retain the Agent
> and
> > app-packages which will then depend on the new hadoop-slider module (in
> > addition to the existing hadoop modules that it already depends on). This
> > will create a single view of Slider codebase, exactly as it stands today,
> > fully supporting the current app-packages. Slider can even release its
> > future versions from this new branch, once the hadoop-sldier module
> > reaches a matured state. It will be an easier path for existing Slider
> > users/app-owners to move to the future stable state of Slider completely
> > off of Hadoop YARN codebase. All that would be expected is to migrate the
> > app-packages to the agent-less version. Of course the Slider community
> > will do the migration for the current app-packages in the Slider
> codebase.
> >
> > -Gour
> >
> > On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vinodkv@apache.org>
> wrote:
> >
> > >Hi, Hadoop YARN community!
> > >
> > >(Cross-posting across Hadoop and Slider communities)
> > >
> > >I opened a JIRA a little while ago to pursue a native YARN framework for
> > >services: https://issues.apache.org/jira/browse/YARN-5079.
> > >
> > >It is part of a bigger effort that a bunch of us YARN community members
> > >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> > >and first-class support for services in YARN.
> > >
> > >The idea is that with our current attention on making services
> > >first­-class, it's time to take a fresh look at how we can make Apache
> > >Hadoop YARN support services well out of the box. I’ve been looking at
> > >various possibilities - ranging from a custom new framework room scratch
> > >to using one of the existing projects - and stopped at Apache Slider
> > >(http://slider.incubator.apache.org) given its association with some of
> > >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > >myself etc.).
> > >
> > >Slider client & AM already handles a great deal of the functionality
> that
> > >we need. I posit that assimilating the client, ApplicationMaster etc of
> > >an existing framework like Apache Slider can serve our purpose really
> > >well. My early informal discussions about this with few Hadoop and
> Slider
> > >community members yielded generally favourable feedback.
> > >
> > >The Apache Slider incubator community also discussed this and expressed
> > >generally positive interest in YARN taking up Slider’s key pieces, you
> > >can see that discussion here: https://s.apache.org/0hoh.
> > >
> > >So in summary, we are looking to the following
> > >
> > >- Code
> > >   ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > >Apache Hadoop for providing a native services experiences in YARN
> > >   ― Leave for now some of the pieces behind in Apache Slider - (a)
> > >Slider agent as we won’t need it, (b) Slider packages that need more
> > >deliberation in terms of where they will live in the long term.
> > >   ― Create a branch in YARN, copy this code over into a new module(s),
> > >and work towards completing a functioning app running on YARN.
> > >
> > >- Communities & releases
> > >   ― Good thing is that many of Apache Slider community members are
> > >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > >without complicating things much, a proposed path forward is active
> > >participation in the branch (as branch committers?) and eventually in
> > >mainline YARN and thus go through a natural progression to committership
> > >/ PMC. Given that most of the members are stalwarts in the Apache
> > >communities, this should be a cinch IMO.
> > >   ― The work on this new code can start, and depending on its state,
> and
> > >assuming that the experiment succeeds, can be merged into trunk and
> later
> > >picked up in the next nearest & feasible Apache Hadoop release.
> > >   ― While the work on forked-over-code goes on till we have a
> > >functioning app, the current Apache Slider project continues to live,
> > >with supporting releases etc. At some point in the future, when things
> > >become clearer, more decisions can be taken on the (parts left behind in
> > >the) incubating project’s future.
> > >
> > >Thoughts?
> > >
> > >Thanks
> > >+Vinod
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message