hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Vasudev <vvasu...@apache.org>
Subject Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider
Date Tue, 19 Jul 2016 10:46:46 GMT
+1. This should go a long way to help running services on YARN.


On 7/19/16, 12:53 AM, "Gour Saha" <gsaha@hortonworks.com> wrote:

>+1 forwarding from the discussion in Slider DL
>Note: On the Slider side, we plan to create a branch corresponding to this
>YARN branch. In this, we intend to remove all the pieces which will move
>to the YARN branch (Slider core, AM, client). We will retain the Agent and
>app-packages which will then depend on the new hadoop-slider module (in
>addition to the existing hadoop modules that it already depends on). This
>will create a single view of Slider codebase, exactly as it stands today,
>fully supporting the current app-packages. Slider can even release its
>future versions from this new branch, once the hadoop-sldier module
>reaches a matured state. It will be an easier path for existing Slider
>users/app-owners to move to the future stable state of Slider completely
>off of Hadoop YARN codebase. All that would be expected is to migrate the
>app-packages to the agent-less version. Of course the Slider community
>will do the migration for the current app-packages in the Slider codebase.
>On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vinodkv@apache.org> wrote:
>>Hi, Hadoop YARN community!
>>(Cross-posting across Hadoop and Slider communities)
>>I opened a JIRA a little while ago to pursue a native YARN framework for
>>services: https://issues.apache.org/jira/browse/YARN-5079.
>>It is part of a bigger effort that a bunch of us YARN community members
>>are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>and first-class support for services in YARN.
>>The idea is that with our current attention on making services
>>first–-class, it's time to take a fresh look at how we can make Apache
>>Hadoop YARN support services well out of the box. I’ve been looking at
>>various possibilities - ranging from a custom new framework room scratch
>>to using one of the existing projects - and stopped at Apache Slider
>>(http://slider.incubator.apache.org) given its association with some of
>>the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>myself etc.).
>>Slider client & AM already handles a great deal of the functionality that
>>we need. I posit that assimilating the client, ApplicationMaster etc of
>>an existing framework like Apache Slider can serve our purpose really
>>well. My early informal discussions about this with few Hadoop and Slider
>>community members yielded generally favourable feedback.
>>The Apache Slider incubator community also discussed this and expressed
>>generally positive interest in YARN taking up Slider’s key pieces, you
>>can see that discussion here: https://s.apache.org/0hoh.
>>So in summary, we are looking to the following
>>- Code
>>   — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>Apache Hadoop for providing a native services experiences in YARN
>>   — Leave for now some of the pieces behind in Apache Slider - (a)
>>Slider agent as we won’t need it, (b) Slider packages that need more
>>deliberation in terms of where they will live in the long term.
>>   — Create a branch in YARN, copy this code over into a new module(s),
>>and work towards completing a functioning app running on YARN.
>>- Communities & releases
>>   — Good thing is that many of Apache Slider community members are
>>already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>without complicating things much, a proposed path forward is active
>>participation in the branch (as branch committers?) and eventually in
>>mainline YARN and thus go through a natural progression to committership
>>/ PMC. Given that most of the members are stalwarts in the Apache
>>communities, this should be a cinch IMO.
>>   — The work on this new code can start, and depending on its state, and
>>assuming that the experiment succeeds, can be merged into trunk and later
>>picked up in the next nearest & feasible Apache Hadoop release.
>>   — While the work on forked-over-code goes on till we have a
>>functioning app, the current Apache Slider project continues to live,
>>with supporting releases etc. At some point in the future, when things
>>become clearer, more decisions can be taken on the (parts left behind in
>>the) incubating project’s future.
>To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message