Mind showing the complete stack trace ?


On Mon, Aug 8, 2016 at 12:30 PM, max square <max2subscribe@gmail.com> wrote:
Thanks Ted for the prompt reply.

There are three or four DFs that are coming from various sources and I'm doing a unionAll on them. 

val placesProcessed = placesUnchanged.unionAll(placesAddedWithMerchantId).unionAll(placesUpdatedFromHotelsWithMerchantId).unionAll(placesUpdatedFromRestaurantsWithMerchantId).unionAll(placesChanged)

I'm using Spark 1.6.2.

On Mon, Aug 8, 2016 at 3:11 PM, Ted Yu <yuzhihong@gmail.com> wrote:
Can you show the code snippet for unionAll operation ?

Which Spark release do you use ?

BTW please use user@spark.apache.org in the future.

On Mon, Aug 8, 2016 at 11:47 AM, max square <max2subscribe@gmail.com> wrote:
Hey guys,

I'm trying to save Dataframe in CSV format after performing unionAll operations on it. 
But I get this exception - 

Exception in thread "main" org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
TungstenExchange hashpartitioning(mId#430,200)

I'm saving it by 

df.write.format("com.databricks.spark.csv").options(Map("mode" -> "DROPMALFORMED", "delimiter" -> "\t", "header" -> "true")).save(bakDir + latest)

It works perfectly if I don't do the unionAll operation.

I see that the format isn't different by printing the part of the results. 

Any help regarding this would be appreciated.