spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Faiz Chachiya <faiz.in...@gmail.com>
Subject Spark DataFrame/DataSet Wide Transformations
Date Thu, 07 Feb 2019 03:20:33 GMT
Hello Team,

With RDDs it is pretty clear which operations would result in wide
transformations and there are also options available to find out parent
dependencies

I have been struggling to do the same with DataFrame/DataSet, I need your
helping in finding out which operations may lead to wide transformations
like (OrderBy) and if there is way to find out the parent dependencies.

There is one way to find out parent dependencies by converting the DF/DS to
RDD and invoke the dependencies.

I hope my question is clear and would request your help with it.

Thanks,
Faiz

Mime
View raw message