spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From karthik padmanabhan <treadston...@gmail.com>
Subject Supporting Apache Aurora as a cluster manager
Date Mon, 11 Sep 2017 03:27:03 GMT
Hi Spark Devs,

We are using Aurora (http://aurora.apache.org/) as our mesos framework for
running stateless services. We would like to use Aurora to deploy big data
and batch workloads as well. And for this we have forked Spark and
implement the ExternalClusterManager trait.

The reason for doing this and not running Spark on Mesos is to leverage the
existing roles and quotas provided by Aurora for admission control and also
leverage Aurora features such as priority and preemption. Additionally we
would like Aurora to be only deploy/orchestration system that our users
should interact with.

We have a working POC where Spark is launching jobs through as the
ClusterManager. Is this something that can be merged upstream ? If so I can
create a design document and create an associated jira ticket.

Thanks
Karthik

Mime
View raw message