spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Cheah (JIRA)" <>
Subject [jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster
Date Fri, 06 Jan 2017 00:48:59 GMT


Matt Cheah commented on SPARK-18278:

I refactored the scheduler code as a thought experiment on what it would take to make the
scheduler pluggable. The goal was to allow writers of a custom scheduler implementation not
need to have any references to {{CoarseGrainedSchedulerBackend}} or {{TaskSchedulerImpl}}.
A preliminary idea for the rewrite for the existing schedulers is [here|],
and it's certainly a non-trivial change. The overarching philosophy of this prototype is to
encourage dependency injection to wire together the scheduler components - see the implementations
of [ExternalClusterManagerFactory|]
like [YarnClusterManagerFactory|]
- but there might be a better way to do this, and I haven't given enough thought to alternate
designs. Some of the new public APIs introduced by this change were also defined arbitrarily
but should be given more careful thought, such as the method signatures on [ExecutorLifecycleHandler|]
and expectations from [ExternalClusterManager.validate()|].
Existing components still refer to {{CoarseGrainedSchedulerBackend}} and {{TaskSchedulerImpl}}
but that's fine since the standalone, YARN, and Mesos scheduler internals should be able to
use the non-public APIs. An implementation of the Kubernetes feature using this draft API
is provided [here|], and the Kubernetes-specific
components don't need to reference {{CoarseGrainedSchedulerBackend}} or {{TaskSchedulerImpl}}.

The thought experiment shows that there would be a non-trivial amount of complexity that would
be introduced if schedulers were to be made truly pluggable. The extra complexity and changes
to the existing scheduler might de-stabilize the existing cluster manager support; for example,
this prototype reorganizes much of the executor loss coordination logic as well but I haven't
tested those changes thoroughly. The alternative to avoid this complexity would be to make
{{CoarseGrainedSchedulerBackend}} and {{TaskSchedulerImpl}} part of the public API, but I'm
extremely wary of going down this path because we would not just be exposing an interface,
but rather we would be exposing a heavily opinionated implementation. Custom subclasses of
{{CoarseGrainedSchedulerBackend}} would be expected to be resilient to changes in the implementations
of these complex classes.

Given the results from my experimentation and the drawbacks I've already highlighted in maintaining
a separate fork, I still would favor integrating Kubernetes into the existing scheduler framework
in-repo and marking it as an experimental feature for several releases, following the precedent
of how the SQL and YARN experimental features were built and released in the past. [~rxin],
where should we go from here?

> Support native submission of spark jobs to a kubernetes cluster
> ---------------------------------------------------------------
>                 Key: SPARK-18278
>                 URL:
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Build, Deploy, Documentation, Scheduler, Spark Core
>            Reporter: Erik Erlandson
>         Attachments: SPARK-18278 - Spark on Kubernetes Design Proposal.pdf
> A new Apache Spark sub-project that enables native support for submitting Spark applications
to a kubernetes cluster.   The submitted application runs in a driver executing on a kubernetes
pod, and executors lifecycles are also managed as pods.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message