spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haripriya Ayyalasomayajula <aharipriy...@gmail.com>
Subject Re: Controlling number of executors on Mesos vs YARN
Date Tue, 11 Aug 2015 13:21:47 GMT
Spark evolved as an example framework for Mesos - thats how I know it. It
is surprising to see that the options provided by mesos in this case are
less. Tweaking the source code, haven't done it yet but I would love to see
what options could be there!

On Tue, Aug 11, 2015 at 5:42 AM, Jerry Lam <chilinglam@gmail.com> wrote:

> My experience with Mesos + Spark is not great. I saw one executor with 30
> CPU and the other executor with 6. So I don't think you can easily
> configure it without some tweaking at the source code.
>
> Sent from my iPad
>
> On 2015-08-11, at 2:38, Haripriya Ayyalasomayajula <aharipriya92@gmail.com>
> wrote:
>
> Hi Tim,
>
> Spark on Yarn allows us to do it using --num-executors and
> --executor_cores commandline arguments. I just got a chance to look at a
> similar spark user list mail, but no answer yet. So does mesos allow
> setting the number of executors and cores? Is there a default number it
> assumes?
>
> On Mon, Jan 5, 2015 at 5:07 PM, Tim Chen <tim@mesosphere.io> wrote:
>
>> Forgot to hit reply-all.
>>
>> ---------- Forwarded message ----------
>> From: Tim Chen <tim@mesosphere.io>
>> Date: Sun, Jan 4, 2015 at 10:46 PM
>> Subject: Re: Controlling number of executors on Mesos vs YARN
>> To: mvle <mvle@us.ibm.com>
>>
>>
>> Hi Mike,
>>
>> You're correct there is no such setting in for Mesos coarse grain mode,
>> since the assumption is that each node is launched with one container and
>> Spark is launching multiple tasks in that container.
>>
>> In fine-grain mode there isn't a setting like that, as it currently will
>> launch an executor as long as it satisfies the minimum container resource
>> requirement.
>>
>> I've created a JIRA earlier about capping the number of executors or
>> better distribute the # of executors launched in each node. Since the
>> decision of choosing what node to launch containers is all in the Spark
>> scheduler side, it's very easy to modify it.
>>
>> Btw, what's the configuration to set the # of executors on YARN side?
>>
>> Thanks,
>>
>> Tim
>>
>>
>>
>> On Sun, Jan 4, 2015 at 9:37 PM, mvle <mvle@us.ibm.com> wrote:
>>
>>> I'm trying to compare the performance of Spark running on Mesos vs YARN.
>>> However, I am having problems being able to configure the Spark workload
>>> to
>>> run in a similar way on Mesos and YARN.
>>>
>>> When running Spark on YARN, you can specify the number of executors per
>>> node. So if I have a node with 4 CPUs, I can specify 6 executors on that
>>> node. When running Spark on Mesos, there doesn't seem to be an equivalent
>>> way to specify this. In Mesos, you can somewhat force this by specifying
>>> the
>>> number of CPU resources to be 6 when running the slave daemon. However,
>>> this
>>> seems to be a static configuration of the Mesos cluster rather something
>>> that can be configured in the Spark framework.
>>>
>>> So here is my question:
>>>
>>> For Spark on Mesos, am I correct that there is no way to control the
>>> number
>>> of executors per node (assuming an idle cluster)? For Spark on Mesos
>>> coarse-grained mode, there is a way to specify max_cores but that is
>>> still
>>> not equivalent to specifying the number of executors per node as when
>>> Spark
>>> is run on YARN.
>>>
>>> If I am correct, then it seems Spark might be at a disadvantage running
>>> on
>>> Mesos compared to YARN (since it lacks the fine tuning ability provided
>>> by
>>> YARN).
>>>
>>> Thanks,
>>> Mike
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
>


-- 
Regards,
Haripriya Ayyalasomayajula

Mime
View raw message