spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Gummelt <mgumm...@mesosphere.io>
Subject Re: Dynamic resource allocation to Spark on Mesos
Date Thu, 02 Feb 2017 21:30:52 GMT
As of Spark 2.0, Mesos mode does support setting cores on the executor
level, but you might need to set the property directly (--conf
spark.executor.cores=<cores>).  I've written about this here:
https://docs.mesosphere.com/1.8/usage/service-guides/spark/job-scheduling/.
That doc is for DC/OS, but the configuration is the same.

On Thu, Feb 2, 2017 at 1:06 PM, Ji Yan <jiyan@drive.ai> wrote:

> I was mainly confused why this is the case with memory, but with cpu
> cores, it is not specified on per executor level
>
> On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt <mgummelt@mesosphere.io>
> wrote:
>
>> It sounds like you've answered your own question, right?
>> --executor-memory means the memory per executor.  If you have no executor
>> w/ 200GB memory, then the driver will accept no offers.
>>
>> On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan <jiyan@drive.ai> wrote:
>>
>>> sorry, to clarify, i was using --executor-memory for memory,
>>> and --total-executor-cores for cpu cores
>>>
>>> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt <mgummelt@mesosphere.io
>>> > wrote:
>>>
>>>> What CLI args are your referring to?  I'm aware of spark-submit's
>>>> arguments (--executor-memory, --total-executor-cores, and --executor-cores)
>>>>
>>>> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan <jiyan@drive.ai> wrote:
>>>>
>>>>> I have done a experiment on this today. It shows that only CPUs are
>>>>> tolerant of insufficient cluster size when a job starts. On my cluster,
I
>>>>> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos
)
>>>>> with --cpu_cores set to 1000, the job starts up with 64 cores. but when
I
>>>>> set --memory to 200Gb, the job fails to start with "Initial job has
>>>>> not accepted any resources; check your cluster UI to ensure that workers
>>>>> are registered and have sufficient resources"
>>>>>
>>>>> Also it is confusing to me that --cpu_cores specifies the number of
>>>>> cpu cores across all executors, but --memory specifies per executor memory
>>>>> requirement.
>>>>>
>>>>> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
>>>>> mgummelt@mesosphere.io> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan <jiyan@drive.ai> wrote:
>>>>>>
>>>>>>> Tasks begin scheduling as soon as the first executor comes up
>>>>>>>
>>>>>>>
>>>>>>> Thanks all for the clarification. Is this the default behavior
of
>>>>>>> Spark on Mesos today? I think this is what we are looking for
because
>>>>>>> sometimes a job can take up lots of resources and later jobs
could not get
>>>>>>> all the resources that it asks for. If a Spark job starts with
only a
>>>>>>> subset of resources that it asks for, does it know to expand
its resources
>>>>>>> later when more resources become available?
>>>>>>>
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Launch each executor with at least 1GB RAM, but if mesos offers
2GB
>>>>>>>> at some moment, then launch an executor with 2GB RAM
>>>>>>>
>>>>>>>
>>>>>>> This is less useful in our use case. But I am also quite interested
>>>>>>> in cases in which this could be helpful. I think this will also
help with
>>>>>>> overall resource utilization on the cluster if when another job
starts up
>>>>>>> that has a hard requirement on resources, the extra resources
to the first
>>>>>>> job can be flexibly re-allocated to the second job.
>>>>>>>
>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
>>>>>>> mgummelt@mesosphere.io> wrote:
>>>>>>>
>>>>>>>> We've talked about that, but it hasn't become a priority
because we
>>>>>>>> haven't had a driving use case.  If anyone has a good argument
for
>>>>>>>> "variable" resource allocation like this, please let me know.
>>>>>>>>
>>>>>>>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin <linshuai2012@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> An alternative behavior is to launch the job with the
best
>>>>>>>>>> resource offer Mesos is able to give
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Michael has just made an excellent explanation about
dynamic
>>>>>>>>> allocation support in mesos. But IIUC, what you want
to achieve is
>>>>>>>>> something like (using RAM as an example) : "Launch each
executor with at
>>>>>>>>> least 1GB RAM, but if mesos offers 2GB at some moment,
then launch an
>>>>>>>>> executor with 2GB RAM".
>>>>>>>>>
>>>>>>>>> I wonder what's benefit of that? To reduce the "resource
>>>>>>>>> fragmentation"?
>>>>>>>>>
>>>>>>>>> Anyway, that is not supported at this moment. In all
the supported
>>>>>>>>> cluster managers of spark (mesos, yarn, standalone, and
the up-to-coming
>>>>>>>>> spark on kubernetes), you have to specify the cores and
memory of each
>>>>>>>>> executor.
>>>>>>>>>
>>>>>>>>> It may not be supported in the future, because only mesos
has the
>>>>>>>>> concepts of offers because of its two-level scheduling
model.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan <jiyan@drive.ai>
wrote:
>>>>>>>>>
>>>>>>>>>> Dear Spark Users,
>>>>>>>>>>
>>>>>>>>>> Currently is there a way to dynamically allocate
resources to
>>>>>>>>>> Spark on Mesos? Within Spark we can specify the CPU
cores, memory before
>>>>>>>>>> running job. The way I understand is that the Spark
job will not run if the
>>>>>>>>>> CPU/Mem requirement is not met. This may lead to
decrease in overall
>>>>>>>>>> utilization of the cluster. An alternative behavior
is to launch the job
>>>>>>>>>> with the best resource offer Mesos is able to give.
Is this possible with
>>>>>>>>>> the current implementation?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Ji
>>>>>>>>>>
>>>>>>>>>> The information in this email is confidential and
may be legally
>>>>>>>>>> privileged. It is intended solely for the addressee.
Access to this email
>>>>>>>>>> by anyone else is unauthorized. If you are not the
intended recipient, any
>>>>>>>>>> disclosure, copying, distribution or any action taken
or omitted to be
>>>>>>>>>> taken in reliance on it, is prohibited and may be
unlawful.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Michael Gummelt
>>>>>>>> Software Engineer
>>>>>>>> Mesosphere
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The information in this email is confidential and may be legally
>>>>>>> privileged. It is intended solely for the addressee. Access to
this email
>>>>>>> by anyone else is unauthorized. If you are not the intended recipient,
any
>>>>>>> disclosure, copying, distribution or any action taken or omitted
to be
>>>>>>> taken in reliance on it, is prohibited and may be unlawful.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Michael Gummelt
>>>>>> Software Engineer
>>>>>> Mesosphere
>>>>>>
>>>>>
>>>>>
>>>>> The information in this email is confidential and may be legally
>>>>> privileged. It is intended solely for the addressee. Access to this email
>>>>> by anyone else is unauthorized. If you are not the intended recipient,
any
>>>>> disclosure, copying, distribution or any action taken or omitted to be
>>>>> taken in reliance on it, is prohibited and may be unlawful.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael Gummelt
>>>> Software Engineer
>>>> Mesosphere
>>>>
>>>
>>>
>>> The information in this email is confidential and may be legally
>>> privileged. It is intended solely for the addressee. Access to this email
>>> by anyone else is unauthorized. If you are not the intended recipient, any
>>> disclosure, copying, distribution or any action taken or omitted to be
>>> taken in reliance on it, is prohibited and may be unlawful.
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>>
>
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>



-- 
Michael Gummelt
Software Engineer
Mesosphere

Mime
View raw message