spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <>
Subject [jira] [Commented] (SPARK-1200) Make it possible to use unmanaged AM in yarn-client mode
Date Thu, 19 Mar 2015 14:01:38 GMT


Steve Loughran commented on SPARK-1200:

You know, we could benefit all YARN apps if the AM launch request could also include a list
of container requests to satisfy. This would remove the pipeline of client->AM->containers,
and start requesting containers earlier. The allocated container list could just come in as
callbacks once the AM is live, as container losses do on AM restart.

It means the client would have to come up with an initial assessment of the priority containers
to escalate. But publishing that information at launch time would help with the (proposed)
Gang scheduling.

> Make it possible to use unmanaged AM in yarn-client mode
> --------------------------------------------------------
>                 Key: SPARK-1200
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 0.9.0
>            Reporter: Sandy Pérez González
>            Assignee: Sandy Ryza
> Using an unmanaged AM in yarn-client mode would allow apps to start up faster, but not
requiring the container launcher AM to be launched on the cluster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message