spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hemant singh <hemant2...@gmail.com>
Subject Re: Spark DataFrame/DataSet Wide Transformations
Date Thu, 07 Feb 2019 04:52:18 GMT
Same concept applies to Dataframe as it is with RDD with respect to
transformations. Both are distributed data set.

Thanks

On Thu, Feb 7, 2019 at 8:51 AM Faiz Chachiya <faiz.india@gmail.com> wrote:

> Hello Team,
>
> With RDDs it is pretty clear which operations would result in wide
> transformations and there are also options available to find out parent
> dependencies
>
> I have been struggling to do the same with DataFrame/DataSet, I need your
> helping in finding out which operations may lead to wide transformations
> like (OrderBy) and if there is way to find out the parent dependencies.
>
> There is one way to find out parent dependencies by converting the DF/DS
> to RDD and invoke the dependencies.
>
> I hope my question is clear and would request your help with it.
>
> Thanks,
> Faiz
>

Mime
View raw message