spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jcdauchy <>
Subject Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?
Date Fri, 04 Oct 2019 15:52:35 GMT
Hello all,

I am surprised that it is not possible to define
"spark.kubernetes.driver.request.cores" when submitting a spark job on

My understanding is that it would limit the CPU requests for the driver on
the k8s cluster and we could still define how many cores (threads) we use in
the driver with spark.driver.cores.

In the 2.4 branch, the request is then defined by "spark.driver.cores", so
we cannot throttle the driver.

Is my understanding wrong or I missed something ?


Sent from:

To unsubscribe e-mail:

View raw message