spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerard Maas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-4940) Document or Support more evenly distributing cores for Mesos mode
Date Mon, 05 Jan 2015 13:19:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264579#comment-14264579
] 

Gerard Maas commented on SPARK-4940:
------------------------------------

>From the perspective of evenly allocating  Spark Streaming consumers (network-bound),
the ideal solution would be to explicitly set the number of hosts.
 
With the current resource allocation policy, we can have eg.  (4),(1),(1) consumers over 3
hosts, instead of the ideal (2),(2),(2). Given that the resource allocation is dynamic at
job startup time, this results in variable performance characteristic for the job being submitted.
  
In practice, we have been restarting the job (using Marathon) until we get a favorable resource
allocation. 

Not sure how well the requirement of a fix amount of executors would fit with the node transparency
offered by Mesos. I'm just trying to elaborate on the requirements from the Spark Streaming
job perspective.

> Document or Support more evenly distributing cores for Mesos mode
> -----------------------------------------------------------------
>
>                 Key: SPARK-4940
>                 URL: https://issues.apache.org/jira/browse/SPARK-4940
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>            Reporter: Timothy Chen
>
> Currently in Coarse grain mode the spark scheduler simply takes all the resources it
can on each node, but can cause uneven distribution based on resources available on each slave.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message