spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-5388) Provide a stable application submission gateway
Date Wed, 28 Jan 2015 23:42:36 GMT


Marcelo Vanzin commented on SPARK-5388:

Hi [~andrewor14],

I read through the spec and the protocol specification seems to be lacking some details. The
mains things that bother me are:

- It's not really a REST API. There's a single endpoint to which you POST different messages.
This sort of forces your hand to use a custom implementation, instead of being able to use
a much nicer framework for this purpose such as JAX-RS. Using a framework like that can later
benefit other parts of Spark too, such as providing a REST API for application data through
the web ui / history server. And as I mentioned in the PR, it allows you to define the endpoints
using classes or interfaces, which serves two purposes: it allows you to do backwards compatibility
checks with tools like MIMA, and it allows you to use the client functionality of JAX-RS for
client requests too (and similar tools for other languages for those who, sort of feeding
back into Dale's comment). Plus, you can use things like Jackson and not care about how to
parse or generate JSON.

- It's unclear how the protocol will be allowed to evolve. What happens when you add a new
field or message in a later version, and that version tries to submit to Spark 1.3? Is there
a version negotiation up front, so that the new client knows to use the old protocol if possible,
or does the client just send the new message and the server will complain if it contains things
it doesn't understand?

The latter kinda feeds into the first comment. With a proper REST-based API, you'd put the
first version of the protocol under "/v1", for example. Later versions are added under "/v2"
and can add new things. Client and server can then negotiate up front (e.g, client needs at
least version "x" for the current app, asks the server for its supported versions, and complains
if "x" is not there).

Also, it could be more specific about how errors are reported. Do you get specific HTTP error
codes for different things? Is there an "Error" type that is sent back to the client in JSON,
and if so, what fields does it have?

> Provide a stable application submission gateway
> -----------------------------------------------
>                 Key: SPARK-5388
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Andrew Or
>            Assignee: Andrew Or
>            Priority: Blocker
>         Attachments: Stable Spark Standalone Submission.pdf
> The existing submission gateway in standalone mode is not compatible across Spark versions.
If you have a newer version of Spark submitting to an older version of the standalone Master,
it is currently not guaranteed to work. The goal is to provide a stable REST interface to
replace this channel.
> The first cut implementation will target standalone cluster mode because there are very
few messages exchanged. The design, however, will be general enough to eventually support
this for other cluster managers too. Note that this is not necessarily required in YARN because
we already use YARN's stable interface to submit applications there.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message