spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <>
Subject Petastorm vs horovod vs tensorflowonspark vs spark_tensorflow_distributor
Date Tue, 01 Jun 2021 21:58:59 GMT
Dear TD, Matei, Michael, Reynold,

I hope all of you and your loved ones are staying safe and doing well.

as a member of the community the direction from the SPARK mentors is
getting to be a bit confusing for me and I was wondering if I can seek your

We have to make long term decisions which is aligned with the open source
SPARK compatibility and directions and it will be wonderful to know what is
the most dependable route to get data from SPARK to tensorflow, is it:
1. petastorm
2. horovod
3. tensorflowonspark
4. spark_tensorflow_distributor
or something else.

Any comments from you will be super useful.

If I am not wrong, seamless integration between SPARK to tensorflow/
pytorch was one of the most exciting visions of SPARK 3.x

While using SPARK ML has its own favourite space, I think that tensorflow
and pytorch will see a lot of focused development as well.

Gourav Sengupta

View raw message