spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <reva...@gmail.com>
Subject Re: GPU Acceleration for spark-3.0.0
Date Thu, 18 Jun 2020 14:05:18 GMT
"So if I am
going to use GPU in my job running on the spark , I still need to code the
map and reduce function in cuda or in c++ and then invoke them throught jni
or something like GPUEnabler , is that right ?"

Sort of.  You could go through all of that work yourself, or you could use
the plugin that we are going to open source in the next few days.  Go to
https://nvidia.com/spark and click on the contact us link.  You should be
able to get the information you want that way. I know from the list of
spark summit talks that others are working on similar things too.  Intel
has a talk about some of their efforts for  columnar processing on FPGAs
and I think SIMD instructions too, at least going off of their talk last
year.

It should be an exciting time for accelerated SQL in spark.



On Wed, Jun 17, 2020 at 11:17 PM charles_cai <1620075790@qq.com> wrote:

> Bobby
>
> Thanks for your answer, it seems that I have misunderstood this paragraph
> in
> the website : *"GPU-accelerate your Apache Spark 3.0 data science
> pipelines—without code changes—and speed up data processing and model
> training while substantially lowering infrastructure costs."* . So if I am
> going to use GPU in my job running on the spark , I still need to code the
> map and reduce function in cuda or in c++ and then invoke them throught jni
> or something like GPUEnabler , is that right ?
>
> thanks
> Charles
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message