spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From charles_cai <1620075...@qq.com>
Subject GPU Acceleration for spark-3.0.0
Date Sat, 13 Jun 2020 03:50:09 GMT
hi,

I have configured the GPU scheduling for spark-3.0.0 on yarn following the
official document ,but the job seems not runing with GPU . Do I need to
modify my code to invoke CUDA ?  Is there any tutorial can be shared ?

running logs:
...
2020-06-13 10:58:01,938 INFO spark.SparkContext: Running Spark version
3.0.0-preview2
2020-06-13 10:58:04,101 INFO resource.ResourceUtils:
==============================================================
2020-06-13 10:58:04,105 INFO resource.ResourceUtils: Resources for
spark.driver:
gpu -> [name: gpu, addresses: 0]


spark-default.conf:
...
spark.executor.resource.gpu.amount  1
spark.worker.resource.gpu.amount    1
spark.driver.resource.gpu.amount    1
spark.driver.resource.gpu.discoveryScript  
/usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh
spark.worker.resource.gpu.discoveryScript  
/usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh
 

nodemanager log:
...
2020-06-13 10:55:07,702 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.ResourcePluginManager:
Found Resource plugins from configuration: [yarn.io/gpu]
2020-06-13 10:55:07,745 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer:
Trying to discover GPU information ...
2020-06-13 10:55:10,601 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer:
Discovered GPU information: === GPUs in the system ===
        Driver Version:440.82
        ProductName=GeForce GTX 950M, MinorNumber=0, TotalMemory=2004MiB,
Utilization=2.0%


Thanks
charles



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message