spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cesar <>
Subject Union of multiple data frames
Date Thu, 05 Apr 2018 18:17:03 GMT
The following code works for small n, but not for large n (>20):

val dfUnion = Seq(df1,df2,df3,...dfn).reduce(_ union _)

By not working, I mean that Spark takes a lot of time to create the
execution plan.

*Is there a more optimal way to perform a union of multiple data frames?*

Cesar Flores

View raw message