spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Shakya <>
Subject Does Apache Spark 3 support GPU usage for Spark RDDs?
Date Tue, 21 Sep 2021 17:38:40 GMT

I am currently trying to run genomic analyses pipelines using Hail(library
for genomics analyses written in python and Scala). Recently, Apache Spark
3 was released and it supported GPU usage.

I tried spark-rapids library to start an on-premise slurm cluster with gpu
nodes. I was able to initialise the cluster. However, when I tried running
hail tasks, the executors kept getting killed.

On querying in Hail forum, I got the response that

That’s a GPU code generator for Spark-SQL, and Hail doesn’t use any
Spark-SQL interfaces, only the RDD interfaces.
So, does Spark3 not support GPU usage for RDD interfaces?

PS: The question is posted in stackoverflow as well: Link


Abhishek Shakya
Senior Data Scientist 1,
Contact: +919002319890 | Email ID:
Aganitha Cognitive Solutions <>

View raw message