spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak Yavuz <brk...@gmail.com>
Subject Re: coalesce on dataFrame
Date Wed, 01 Jul 2015 17:04:58 GMT
You can use df.repartition(1) in Spark 1.4. See here
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L1396>
.

Best,
Burak

On Wed, Jul 1, 2015 at 3:05 AM, Olivier Girardot <ssaboum@gmail.com> wrote:

> PySpark or Spark (scala) ?
> When you use coalesce with anything but a column you must use a literal
> like that in PySpark :
>
> from pyspark.sql import functions as F
>
> F.coalesce(df.a, F.lit(True))
>
> Le mer. 1 juil. 2015 à 12:03, Ewan Leith <ewan.leith@realitymine.com> a
> écrit :
>
>> It's in spark 1.4.0, or should be at least:
>>
>> https://issues.apache.org/jira/browse/SPARK-6972
>>
>> Ewan
>>
>> -----Original Message-----
>> From: Hafiz Mujadid [mailto:hafizmujadid00@gmail.com]
>> Sent: 01 July 2015 08:23
>> To: user@spark.apache.org
>> Subject: coalesce on dataFrame
>>
>> How can we use coalesce(1, true) on dataFrame?
>>
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
>> commands, e-mail: user-help@spark.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>

Mime
View raw message