spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kimoon Kim <kim...@pepperdata.com>
Subject Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods
Date Fri, 30 Mar 2018 18:40:04 GMT
> Instead of requesting `[driver,executor].memory`, we should just request
`[driver,executor].memory + [driver,executor].memoryOverhead `. I think
this case is a bit clearer than the CPU case, so I went ahead and filed an
issue <https://issues.apache.org/jira/browse/SPARK-23825> with more details
and made a PR <https://github.com/apache/spark/pull/20943>.

I think this suggestion makes sense.

> One way to solve this could be to request more than 1 core from
Kubernetes per task. The exact amount we should request is unclear to me
(it largely depends on how many threads actually get spawned for a task).

I wonder if this is being addressed by PR #20553
<https://github.com/apache/spark/pull/20553> written by Yinan. Yinan?

Thanks,
Kimoon

On Thu, Mar 29, 2018 at 5:14 PM, David Vogelbacher <
dvogelbacher@palantir.com> wrote:

> Hi,
>
>
>
> At the moment driver and executor pods are created using the following
> requests and limits:
>
>
>
> *CPU*
>
> *Memory*
>
> *Request*
>
> [driver,executor].cores
>
> [driver,executor].memory
>
> *Limit*
>
> Unlimited (but can be specified using spark.[driver,executor].cores)
>
> [driver,executor].memory + [driver,executor].memoryOverhead
>
>
>
> Specifying the requests like this leads to problems if the pods only get
> the requested amount of resources and nothing of the optional (limit)
> resources, as it can happen in a fully utilized cluster.
>
>
>
> *For memory:*
>
> Let’s say we have a node with 100GiB memory and 5 pods with 20 GiB memory
> and 5 GiB memoryOverhead.
>
> At the beginning all 5 pods use 20 GiB of memory and all is well. If a pod
> then starts using its overhead memory it will get killed as there is no
> more memory available, even though we told spark
>
> that it can use 25 GiB of memory.
>
>
>
> Instead of requesting `[driver,executor].memory`, we should just request
> `[driver,executor].memory + [driver,executor].memoryOverhead `.
>
> I think this case is a bit clearer than the CPU case, so I went ahead and
> filed an issue <https://issues.apache.org/jira/browse/SPARK-23825> with
> more details and made a PR <https://github.com/apache/spark/pull/20943>.
>
>
>
> *For CPU:*
>
> As it turns out, there can be performance problems if we only have
> `executor.cores` available (which means we have one core per task). This
> was raised here <https://github.com/apache-spark-on-k8s/spark/issues/352>
> and is the reason that the cpu limit was set to unlimited.
>
> This issue stems from the fact that in general there will be more than one
> thread per task, resulting in performance impacts if there is only one core
> available.
>
> However, I am not sure that just setting the limit to unlimited is the
> best solution because it means that even if the Kubernetes cluster can
> perfectly satisfy the resource requests, performance might be very bad.
>
>
>
> I think we should guarantee that an executor is able to do its work well
> (without performance issues or getting killed - as could happen in the
> memory case) with the resources it gets guaranteed from Kubernetes.
>
>
>
> One way to solve this could be to request more than 1 core from Kubernetes
> per task. The exact amount we should request is unclear to me (it largely
> depends on how many threads actually get spawned for a task).
>
> We would need to find a way to determine this somehow automatically or at
> least come up with a better default value than 1 core per task.
>
>
>
> Does somebody have ideas or thoughts on how to solve this best?
>
>
>
> Best,
>
> David
>

Mime
View raw message