spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From max square <max2subscr...@gmail.com>
Subject Re: Getting a TreeNode Exception while saving into Hadoop
Date Mon, 08 Aug 2016 19:30:31 GMT
Thanks Ted for the prompt reply.

There are three or four DFs that are coming from various sources and I'm
doing a unionAll on them.

val placesProcessed = placesUnchanged.unionAll(placesAddedWithMerchantId
).unionAll(placesUpdatedFromHotelsWithMerchantId).unionAll(
placesUpdatedFromRestaurantsWithMerchantId).unionAll(placesChanged)

I'm using Spark 1.6.2.

On Mon, Aug 8, 2016 at 3:11 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you show the code snippet for unionAll operation ?
>
> Which Spark release do you use ?
>
> BTW please use user@spark.apache.org in the future.
>
> On Mon, Aug 8, 2016 at 11:47 AM, max square <max2subscribe@gmail.com>
> wrote:
>
>> Hey guys,
>>
>> I'm trying to save Dataframe in CSV format after performing unionAll
>> operations on it.
>> But I get this exception -
>>
>> Exception in thread "main" org.apache.spark.sql.catalyst.
>> errors.package$TreeNodeException: execute, tree:
>> TungstenExchange hashpartitioning(mId#430,200)
>>
>> I'm saving it by
>>
>> df.write.format("com.databricks.spark.csv").options(Map("mode" ->
>> "DROPMALFORMED", "delimiter" -> "\t", "header" -> "true")).save(bakDir +
>> latest)
>>
>> It works perfectly if I don't do the unionAll operation.
>> I see that the format isn't different by printing the part of the
>> results.
>>
>> Any help regarding this would be appreciated.
>>
>>
>

Mime
View raw message