spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: [SQL] A bug with withColumn?
Date Thu, 31 Mar 2016 21:47:39 GMT
Hi,

Thanks Ted.

It means that it's not only possible to rename a column using
withColumnRenamed, but also replace the content of a column (in one
shot) using withColumn with an existing column name. I can live with
that :)

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Mar 31, 2016 at 5:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> Looks like this is result of the following check:
>
>     val shouldReplace = output.exists(f => resolver(f.name, colName))
>     if (shouldReplace) {
>
> where existing column, text, was replaced.
>
> On Thu, Mar 31, 2016 at 12:08 PM, Jacek Laskowski <jacek@japila.pl> wrote:
>>
>> Hi,
>>
>> Just ran into the following. Is this a bug?
>>
>> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT2",
>> lit(5)).show
>> +---+-------+---+-----+-----+
>> | id|   text| id| text|TEXT2|
>> +---+-------+---+-----+-----+
>> |  0|  hello|  0|  two|    5|
>> |  1|swiecie|  1|three|    5|
>> +---+-------+---+-----+-----+
>>
>>
>> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT",
>> lit(5)).show
>> +---+----+---+----+
>> | id|TEXT| id|TEXT|
>> +---+----+---+----+
>> |  0|   5|  0|   5|
>> |  1|   5|  1|   5|
>> +---+----+---+----+
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message