spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Nielsen <nik...@mesosphere.io>
Subject Re: Spark on Mesos vs Yarn
Date Wed, 10 Jun 2015 04:51:09 GMT
Hi Bharath (and rest of Spark dev list!),

Just a small shout out: I am a Apache Mesos Committer and would love to
help out with anything you need to get this going.

Cheers,
Nik

On 9 June 2015 at 21:10, Bharath Ravi Kumar <reachbach@gmail.com> wrote:

> All,
>
> Despite the common origin of spark & mesos, the stability and adoption of
> mesos, and the age of the spark-mesos binding, I find the mesos support
> less mature, with fundamental shortcomings (like framework auth
> <https://issues.apache.org/jira/browse/SPARK-6284>) remaining unresolved.
> If there's shortage of developer time, I'd be glad to contribute, but it's
> unclear if the committer group has sufficient time (and priority) to take
> the mesos support forward. While it has been stated often that support for
> mesos & yarn are equally important, that doesn't seem to translate to
> visible progress. I'd be glad if my observation is incorrect as I seek
> better focus and long term commitment on the mesos support.
> As for the specific issue (6284), I'm happy to build, testing & eventually
> deploy the patch in our production cluster, but I'd rather see it becoming
> mainstream.
> Thanks for your consideration.
>
> -Bharath
>
>
> On Thu, May 28, 2015 at 9:18 AM, Bharath Ravi Kumar <reachbach@gmail.com>
> wrote:
>
>> A follow up : considering that spark on mesos is indeed important to
>> databricks, its partners and the community, fundamental issues like
>> spark-6284 shouldn't be languishing for this long. A mesos cluster hosting
>> diverse (i.e.multi-tenant)  workloads is a common scenario in production
>> for serious users. The ability to auth a framework & assign roles would be
>> a fairly basic ask, one would imagine. Is the lack of time / effort a
>> constraint? If so, I'd be glad to help (as mentioned in the jira).
>>
>> On Fri, May 15, 2015 at 5:29 PM, Iulian DragoČ™ <
>> iulian.dragos@typesafe.com> wrote:
>>
>>> Hi Ankur,
>>>
>>> Just to add a thought to Tim's excellent answer, Spark on Mesos is very
>>> important to us and is the recommended deployment for our customers as
>>> Typesafe.
>>>
>>> Thanks for pointing to your PR, I see Tim already went through a round
>>> of reviews. It seems very useful, I'll give it a try as well.
>>>
>>> thanks,
>>> iulian
>>>
>>>
>>>
>>> On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan <ankur@malloc64.com>
>>> wrote:
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> Hi Tim,
>>>>
>>>> Thanks for such a detailed email. I am excited to hear about the new
>>>> features, I had a pull request going for adding "attribute based
>>>> filtering in the mesos scheduler" but it hasn't received much love -
>>>> https://github.com/apache/spark/pull/5563 . I am a fan of
>>>> mesos/marathon/mesosphere and spark ecosystems and trying to push
>>>> adoption at my workplace.
>>>>
>>>> It would love to see documentation, tutorials (anything actually) that
>>>> would make mesos + spark a better and more fleshed out solution. Would
>>>> it be possible for you to share some links to the JIRA and pull
>>>> requests so that I can keep track on the progress/features.
>>>>
>>>> Again, thanks for replying.
>>>>
>>>> - -- Ankur Chauhan
>>>>
>>>> On 15/05/2015 00:39, Tim Chen wrote:
>>>> > Hi Ankur,
>>>> >
>>>> > This is a great question as I've heard similar concerns about Spark
>>>> > on Mesos.
>>>> >
>>>> > At the time when I started to contribute to Spark on Mesos approx
>>>> > half year ago, the Mesos scheduler and related code hasn't really
>>>> > got much attention from anyone and it was pretty much in
>>>> > maintenance mode.
>>>> >
>>>> > As a Mesos PMC that is really interested in Spark I started to
>>>> > refactor and check out different JIRAs and PRs around the Mesos
>>>> > scheduler, and after that started to fix various bugs in Spark,
>>>> > added documentation and also in fix related Mesos issues as well.
>>>> >
>>>> > Just recently for 1.4 we've merged in Cluster mode and Docker
>>>> > support, and there are also pending PRs around framework
>>>> > authentication, multi-role support, dynamic allocation, more finer
>>>> > tuned coarse grain mode scheduling configurations, etc.
>>>> >
>>>> > And finally just want to mention that Mesosphere and Typesafe is
>>>> > collaborating to bring a certified distribution
>>>> > (
>>>> https://databricks.com/spark/certification/certified-spark-distributi
>>>> on)
>>>> > of Spark on Mesos and DCOS, and we will be pouring resources into
>>>> > not just maintain Spark on Mesos but drive more features into the
>>>> > Mesos scheduler and also in Mesos so stateful services can leverage
>>>> > new APIs and features to make better scheduling decisions and
>>>> > optimizations.
>>>> >
>>>> > I don't have a solidified roadmap to share yet, but we will be
>>>> > discussing this and hopefully can share with the community soon.
>>>> >
>>>> > In summary Spark on Mesos is not dead or in maintenance mode, and
>>>> > look forward to see a lot more changes from us and the community.
>>>> >
>>>> > Tim
>>>> >
>>>> > On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
>>>> > <ankur@malloc64.com <mailto:ankur@malloc64.com>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > This is both a survey type as well as a roadmap query question. It
>>>> > seems like of the cluster options to run spark (i.e. via YARN and
>>>> > Mesos), YARN seems to be getting a lot more attention and patches
>>>> > when compared to Mesos.
>>>> >
>>>> > Would it be correct to assume that spark on mesos is more or less
>>>> > a dead or something like a maintenance-only feature and YARN is
>>>> > the recommended way to go?
>>>> >
>>>> > What is the roadmap for spark on mesos? and what is the roadmap
>>>> > for spark on yarn. I like mesos so as much as I would like to see
>>>> > it thrive I don't think spark community is active (or maybe it
>>>> > just appears that way).
>>>> >
>>>> > Another more community oriented question: what do most people use
>>>> > to run spark in production or more-than-POC products? Why did you
>>>> > make that decision?
>>>> >
>>>> > There was a similar post form early 2014 where Metei answered that
>>>> > mesos and yarn were equally important, but has this changed as
>>>> > spark has now reached almost 1.4.0 stage?
>>>> >
>>>> > -- Ankur Chauhan
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> >
>>>> >
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> > <mailto:user-unsubscribe@spark.apache.org> For additional commands,
>>>> > e-mail: user-help@spark.apache.org
>>>> > <mailto:user-help@spark.apache.org>
>>>> >
>>>> >
>>>> -----BEGIN PGP SIGNATURE-----
>>>>
>>>> iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
>>>> xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
>>>> PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
>>>> A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
>>>> tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
>>>> w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
>>>> =4Aax
>>>> -----END PGP SIGNATURE-----
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> --
>>> Iulian Dragos
>>>
>>> ------
>>> Reactive Apps on the JVM
>>> www.typesafe.com
>>>
>>>
>>
>

Mime
View raw message