My experience with Mesos + Spark is not great. I saw one executor with 30 CPU and the other executor with 6. So I don't think you can easily configure it without some tweaking at the source code.

Sent from my iPad

On 2015-08-11, at 2:38, Haripriya Ayyalasomayajula <aharipriya92@gmail.com> wrote:

Hi Tim,

Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores? Is there a default number it assumes?

On Mon, Jan 5, 2015 at 5:07 PM, Tim Chen <tim@mesosphere.io> wrote:
Forgot to hit reply-all.

---------- Forwarded message ----------
From: Tim Chen <tim@mesosphere.io>
Date: Sun, Jan 4, 2015 at 10:46 PM
Subject: Re: Controlling number of executors on Mesos vs YARN
To: mvle <mvle@us.ibm.com>


Hi Mike,

You're correct there is no such setting in for Mesos coarse grain mode, since the assumption is that each node is launched with one container and Spark is launching multiple tasks in that container.

In fine-grain mode there isn't a setting like that, as it currently will launch an executor as long as it satisfies the minimum container resource requirement.

I've created a JIRA earlier about capping the number of executors or better distribute the # of executors launched in each node. Since the decision of choosing what node to launch containers is all in the Spark scheduler side, it's very easy to modify it.

Btw, what's the configuration to set the # of executors on YARN side?

Thanks,

Tim



On Sun, Jan 4, 2015 at 9:37 PM, mvle <mvle@us.ibm.com> wrote:
I'm trying to compare the performance of Spark running on Mesos vs YARN.
However, I am having problems being able to configure the Spark workload to
run in a similar way on Mesos and YARN.

When running Spark on YARN, you can specify the number of executors per
node. So if I have a node with 4 CPUs, I can specify 6 executors on that
node. When running Spark on Mesos, there doesn't seem to be an equivalent
way to specify this. In Mesos, you can somewhat force this by specifying the
number of CPU resources to be 6 when running the slave daemon. However, this
seems to be a static configuration of the Mesos cluster rather something
that can be configured in the Spark framework.

So here is my question:

For Spark on Mesos, am I correct that there is no way to control the number
of executors per node (assuming an idle cluster)? For Spark on Mesos
coarse-grained mode, there is a way to specify max_cores but that is still
not equivalent to specifying the number of executors per node as when Spark
is run on YARN.

If I am correct, then it seems Spark might be at a disadvantage running on
Mesos compared to YARN (since it lacks the fine tuning ability provided by
YARN).

Thanks,
Mike



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org






--
Regards,
Haripriya Ayyalasomayajula