hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi Ozawa <oz...@apache.org>
Subject Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider
Date Tue, 19 Jul 2016 03:43:28 GMT
+1,

>From user's point of view, the high-level resource management API
without any additional dependency is much comfortable than using
low-level APIs. I believe the way should increase YARN applications
including long-running services. From developer's point of view, the
integration make YARN more stable and feature-rich since we can get
feedback about YARN low-level APIs including backward compatibility,
features we need, and so on.

BTW, I have a question about language-bindings: can we prepare python,
C++, or another language APIs over Slider? Is it out of focus of this
discussion?

Thanks,
- Tsuyoshi

On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli
<vinodkv@apache.org> wrote:
> Hi, Hadoop YARN community!
>
> (Cross-posting across Hadoop and Slider communities)
>
> I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079.
>
> It is part of a bigger effort that a bunch of us YARN community members are interested
in making progress on: YARN-4692 - [Umbrella] Simplified and first-class support for services
in YARN.
>
> The idea is that with our current attention on making services first­-class, it's time
to take a fresh look at how we can make Apache Hadoop YARN support services well out of the
box. I’ve been looking at various possibilities - ranging from a custom new framework room
scratch to using one of the existing projects - and stopped at Apache Slider (http://slider.incubator.apache.org)
given its association with some of the YARN community members (Steve Loughran, Devaraj Das,
Arun C Murthy, myself etc.).
>
> Slider client & AM already handles a great deal of the functionality that we need.
I posit that assimilating the client, ApplicationMaster etc of an existing framework like
Apache Slider can serve our purpose really well. My early informal discussions about this
with few Hadoop and Slider community members yielded generally favourable feedback.
>
> The Apache Slider incubator community also discussed this and expressed generally positive
interest in YARN taking up Slider’s key pieces, you can see that discussion here: https://s.apache.org/0hoh.
>
> So in summary, we are looking to the following
>
> - Code
>    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache Hadoop
for providing a native services experiences in YARN
>    — Leave for now some of the pieces behind in Apache Slider - (a) Slider agent as
we won’t need it, (b) Slider packages that need more deliberation in terms of where they
will live in the long term.
>    — Create a branch in YARN, copy this code over into a new module(s), and work towards
completing a functioning app running on YARN.
>
> - Communities & releases
>    — Good thing is that many of Apache Slider community members are already seasoned
folk in the Apache Hadoop ecosystem projects. For those committers & PMC in Slider that
are not yet Hadoop committers / PMC, without complicating things much, a proposed path forward
is active participation in the branch (as branch committers?) and eventually in mainline YARN
and thus go through a natural progression to committership / PMC. Given that most of the members
are stalwarts in the Apache communities, this should be a cinch IMO.
>    — The work on this new code can start, and depending on its state, and assuming
that the experiment succeeds, can be merged into trunk and later picked up in the next nearest
& feasible Apache Hadoop release.
>    — While the work on forked-over-code goes on till we have a functioning app, the
current Apache Slider project continues to live, with supporting releases etc. At some point
in the future, when things become clearer, more decisions can be taken on the (parts left
behind in the) incubating project’s future.
>
> Thoughts?
>
> Thanks
> +Vinod
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message