spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cesar <ces...@gmail.com>
Subject Union of multiple data frames
Date Thu, 05 Apr 2018 18:17:03 GMT
The following code works for small n, but not for large n (>20):

val dfUnion = Seq(df1,df2,df3,...dfn).reduce(_ union _)
dfUnion.show()

By not working, I mean that Spark takes a lot of time to create the
execution plan.

*Is there a more optimal way to perform a union of multiple data frames?*


thanks
-- 
Cesar Flores

Mime
View raw message