spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cfangmac <cfang1...@aliyun.com.INVALID>
Subject FW: Spark3.0 gpu support
Date Mon, 09 Dec 2019 08:16:31 GMT
 

 

 

 

 

发件人: cfangmac <cfang1109@aliyun.com>
日期: 2019年12月9日 星期一 下午3:49
收件人: <dev@spark.apache.org>
主题: Spark3.0 gpu support

 

Hi everyone,

 

Recently I use the master branch of Apache Spark from github and try to use the function of
GPU-aware scheduling.

 

I setup a standalone cluster and set some GPU related config options,such as,

A) spark.worker.resourceFile,which is followd by a json format file that contains gpu
addresses;

B) spark.worker.resource.gpu.amount, which specified the gpu amount for each worker;

C) spark.executor.resource.gpu.amount, which specified the gpu amount for each executor;

D)spark.task.resource.gpu.amount, which specified the gpu request from each task;

 

Then I run a k-means training program which I thought would require many mathematical operations
and gpu is thought to be helpful to accelerate the training. I got the web page as follow
and it seems those gpu options are configured correctly, however I used the gpu monitor tool
and found that those gpus  seems does not be used, that is to say the training program is
still run on cpu other than gpu.

 

Now I am confused about two points,

1, is there something I missed that caused the fail to use gpu?

2, After the task is deserialized in executor, how does a jvm(Java/Scala) program run on
gpu?

Does the spark executor use JNI + cuda/opencl or other tools? 

 

 

Thanks,

Chao Fang

 


Mime
View raw message