spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hemant singh <>
Subject Re: Spark DataFrame/DataSet Wide Transformations
Date Thu, 07 Feb 2019 04:52:18 GMT
Same concept applies to Dataframe as it is with RDD with respect to
transformations. Both are distributed data set.


On Thu, Feb 7, 2019 at 8:51 AM Faiz Chachiya <> wrote:

> Hello Team,
> With RDDs it is pretty clear which operations would result in wide
> transformations and there are also options available to find out parent
> dependencies
> I have been struggling to do the same with DataFrame/DataSet, I need your
> helping in finding out which operations may lead to wide transformations
> like (OrderBy) and if there is way to find out the parent dependencies.
> There is one way to find out parent dependencies by converting the DF/DS
> to RDD and invoke the dependencies.
> I hope my question is clear and would request your help with it.
> Thanks,
> Faiz

View raw message