hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider
Date Wed, 20 Jul 2016 22:37:45 GMT
An interesting proposal, we can continue this discussion on the Slider dev lists.

Thanks
+Vinod

> On Jul 19, 2016, at 4:51 PM, jonmaron@gmail.com wrote:
> 
> +1 - this approach definitely addresses an important need.
> 
> The project has gone through at least a couple of development/repo
> approaches:
> 
> 1)  git-flow with master/develop branches
> 2)  abandonment of the master branch and work/releases based on the develop
> branch.
> 
> I have no strong objections to the approach above, but given the existence
> of the two branches above, and some developer familiarity with the
> 'develop' branch, perhaps we could:
> 
> 1)  Synch the master branch to the develop branch (not sure which git
> mechanism would best achieve this)
> 2)  Leverage the develop branch to create the agent/app-package based
> Slider distribution
> 
> In this way there'a a legacy branch available while develop continues to
> proceed on the existing and familiar development branch.
> 
> Just a thought.  Like I said - no real strong feelings :)
> 
> On Mon, Jul 18, 2016 at 3:23 PM, Gour Saha <gsaha@hortonworks.com> wrote:
> 
>> +1 forwarding from the discussion in Slider DL
>> 
>> Note: On the Slider side, we plan to create a branch corresponding to this
>> YARN branch. In this, we intend to remove all the pieces which will move
>> to the YARN branch (Slider core, AM, client). We will retain the Agent and
>> app-packages which will then depend on the new hadoop-slider module (in
>> addition to the existing hadoop modules that it already depends on). This
>> will create a single view of Slider codebase, exactly as it stands today,
>> fully supporting the current app-packages. Slider can even release its
>> future versions from this new branch, once the hadoop-sldier module
>> reaches a matured state. It will be an easier path for existing Slider
>> users/app-owners to move to the future stable state of Slider completely
>> off of Hadoop YARN codebase. All that would be expected is to migrate the
>> app-packages to the agent-less version. Of course the Slider community
>> will do the migration for the current app-packages in the Slider codebase.
>> 
>> -Gour
>> 
>> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vinodkv@apache.org> wrote:
>> 
>>> Hi, Hadoop YARN community!
>>> 
>>> (Cross-posting across Hadoop and Slider communities)
>>> 
>>> I opened a JIRA a little while ago to pursue a native YARN framework for
>>> services: https://issues.apache.org/jira/browse/YARN-5079.
>>> 
>>> It is part of a bigger effort that a bunch of us YARN community members
>>> are interested in making progress on: YARN-4692 - [Umbrella] Simplified
>>> and first-class support for services in YARN.
>>> 
>>> The idea is that with our current attention on making services
>>> first­-class, it's time to take a fresh look at how we can make Apache
>>> Hadoop YARN support services well out of the box. I’ve been looking at
>>> various possibilities - ranging from a custom new framework room scratch
>>> to using one of the existing projects - and stopped at Apache Slider
>>> (http://slider.incubator.apache.org) given its association with some of
>>> the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
>>> myself etc.).
>>> 
>>> Slider client & AM already handles a great deal of the functionality that
>>> we need. I posit that assimilating the client, ApplicationMaster etc of
>>> an existing framework like Apache Slider can serve our purpose really
>>> well. My early informal discussions about this with few Hadoop and Slider
>>> community members yielded generally favourable feedback.
>>> 
>>> The Apache Slider incubator community also discussed this and expressed
>>> generally positive interest in YARN taking up Slider’s key pieces, you
>>> can see that discussion here: https://s.apache.org/0hoh.
>>> 
>>> So in summary, we are looking to the following
>>> 
>>> - Code
>>>  ― ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
>>> Apache Hadoop for providing a native services experiences in YARN
>>>  ― Leave for now some of the pieces behind in Apache Slider - (a)
>>> Slider agent as we won’t need it, (b) Slider packages that need more
>>> deliberation in terms of where they will live in the long term.
>>>  ― Create a branch in YARN, copy this code over into a new module(s),
>>> and work towards completing a functioning app running on YARN.
>>> 
>>> - Communities & releases
>>>  ― Good thing is that many of Apache Slider community members are
>>> already seasoned folk in the Apache Hadoop ecosystem projects. For those
>>> committers & PMC in Slider that are not yet Hadoop committers / PMC,
>>> without complicating things much, a proposed path forward is active
>>> participation in the branch (as branch committers?) and eventually in
>>> mainline YARN and thus go through a natural progression to committership
>>> / PMC. Given that most of the members are stalwarts in the Apache
>>> communities, this should be a cinch IMO.
>>>  ― The work on this new code can start, and depending on its state, and
>>> assuming that the experiment succeeds, can be merged into trunk and later
>>> picked up in the next nearest & feasible Apache Hadoop release.
>>>  ― While the work on forked-over-code goes on till we have a
>>> functioning app, the current Apache Slider project continues to live,
>>> with supporting releases etc. At some point in the future, when things
>>> become clearer, more decisions can be taken on the (parts left behind in
>>> the) incubating project’s future.
>>> 
>>> Thoughts?
>>> 
>>> Thanks
>>> +Vinod
>> 
>> 
> 
> 
> -- 
> Sent from some random computer....


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message