spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <>
Subject [jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode
Date Thu, 05 Feb 2015 03:02:35 GMT


Andrew Or commented on SPARK-5388:

Hi [~vanzin], thank you for all of your comments. I agree with many of the points that you
raised and have addressed them in the PR. I'd just like to clarify, in case there is any confusion,
that this is intended to be an internal submission gateway only for standalone cluster mode.
This is in no way expected to interface with third party applications or intended to be a
stable alternative to Spark submit across all cluster managers and deploy modes

For this reason, the protocol has very few core requirements, the most important of which,
backward and forward compatibility, is sufficiently handled by using any stable RPC framework.
It's just so that we chose a HTTP/JSON-based design because it is the simplest one that fulfills
this requirement. The initial design did not adhere closely to conventions commonly used by
other REST protocols because this was not strictly necessary to achieve the compatibility
guarantees we wanted to provide. Not to mention there is not really a formal W3C-style REST
specification, and many existing REST protocols out there (e.g. twitter, flickr, github) actually
diverge widely in terms of the extent to which they adherence to these vaguely-defined conventions.

That said, I agree completely that the initial design doc was vague. I have updated it by
expanding in much greater detail on things like how the server responds to different error
conditions and how versioning is handled. Please have a look and let me know if you have any
other concerns.

> Provide a stable application submission gateway in standalone cluster mode
> --------------------------------------------------------------------------
>                 Key: SPARK-5388
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Andrew Or
>            Assignee: Andrew Or
>            Priority: Blocker
>         Attachments: stable-spark-submit-in-standalone-mode-2-4-15.pdf
> The existing submission gateway in standalone mode is not compatible across Spark versions.
If you have a newer version of Spark submitting to an older version of the standalone Master,
it is currently not guaranteed to work. The goal is to provide a stable REST interface to
replace this channel.
> The first cut implementation will target standalone cluster mode because there are very
few messages exchanged. The design, however, should be general enough to potentially support
this for other cluster managers too. Note that this is not necessarily required in YARN because
we already use YARN's stable interface to submit applications there.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message