spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Melo <andrew.m...@gmail.com>
Subject Re: Detect executor core count
Date Tue, 18 Jun 2019 22:47:22 GMT
On Tue, Jun 18, 2019 at 5:40 PM Steve Loughran <stevel@cloudera.com.invalid>
wrote:

> be aware that older java 8 versions count the #of cores in the host, not
> those allocated for the container they run in
> https://bugs.openjdk.java.net/browse/JDK-8140793
>
>
Ergh, that's good to know. I suppose, though, that in any case, there
should be a SparkSession available if I'm in the executor context, so I can
fallback to something sensible just in case.

Thanks for the help, everyone!


> On Tue, Jun 18, 2019 at 8:13 PM Ilya Matiach <ilmat@microsoft.com.invalid>
> wrote:
>
>> Hi Andrew,
>>
>> I tried to do something similar to that in the LightGBM
>> classifier/regressor/ranker in mmlspark package, I try to use the spark
>> conf and if not configured I get the processors from the JVM directly:
>>
>>
>> https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils.scala#L172
>>
>>
>>
>> If you know of a better way, please let me know!
>>
>>
>>
>>     val spark = dataset.sparkSession
>>
>>     try {
>>
>>       val confCores = spark.sparkContext.getConf
>>
>>         .get("spark.executor.cores").toInt
>>
>>       val confTaskCpus = spark.sparkContext.getConf
>>
>>         .get("spark.task.cpus", "1").toInt
>>
>>       confCores / confTaskCpus
>>
>>     } catch {
>>
>>       case _: NoSuchElementException =>
>>
>>         // If spark.executor.cores is not defined, get the cores per JVM
>>
>>         import spark.implicits._
>>
>>         val numMachineCores = spark.range(0, 1)
>>
>>           .map(_ =>
>> java.lang.Runtime.getRuntime.availableProcessors).collect.head
>>
>>         numMachineCores
>>
>>     }
>>
>>
>>
>> Thank you, Ilya
>>
>>
>>
>> *From:* Andrew Melo <andrew.melo@gmail.com>
>> *Sent:* Tuesday, June 18, 2019 11:32 AM
>> *To:* dev <dev@spark.apache.org>
>> *Subject:* Detect executor core count
>>
>>
>>
>> Hello,
>>
>>
>>
>> Is there a way to detect the number of cores allocated for an executor
>> within a java-based InputPartitionReader?
>>
>>
>>
>> Thanks!
>>
>> Andrew
>>
>

Mime
View raw message