spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Spark DF does not rename the column
Date Mon, 04 Jan 2021 18:05:43 GMT
Hi,

 version 2.4.3

I don't know the cause of this.

This renaming of DF columns used to work fine. I did couple of changes to
spark/Scala code not relevant to this table and it refuses to rename the
columns for a table!.

val summaryACC = HiveContext.table("summaryACC")

summaryACC.printSchema()

root
 |-- ceated: string (nullable = true)
 |-- hashtag: string (nullable = true)
 |-- paid: float (nullable = true)
 |-- received: float (nullable = true)

summaryACC.
    orderBy(desc("paid"),desc("received")).
    withColumnRenamed("created","Date Calculated").
    withColumnRenamed("hashtag","Who").
    withColumn(("received"),format_number(col("received"),2)).
    withColumn(("paid"),format_number(col("paid"),2)).
    withColumnRenamed("paid","paid out/GBP").
    withColumnRenamed("received","paid in/GBP").
    withColumn("paid in/GBP",when(col("paid in/GBP") ===
"0.00","--").otherwise(col("paid in/GBP"))).
    withColumn("paid out/GBP",when(col("paid out/GBP") ===
"0.00","--").otherwise(col("paid out/GBP"))).
    select("Date Calculated","Who","paid in/GBP","paid
out/GBP").show(1000,false)

and this is the error

org.apache.spark.sql.AnalysisException: cannot resolve '`Date Calculated`'
given input columns: [alayer.summaryacc.ceated, Who, paid out/GBP, paid
in/GBP];;


This used to work before!


+----------------------------+------------------+-----------+------------+

|Date Calculated             |Who               |paid in/GBP|paid out/GBP|

+----------------------------+------------------+-----------+------------+

|Mon Jan 04 14:22:17 GMT 2021|paypal            |579.98     |1,526.86    |


Appreciate any ideas.


Thanks, Mich

Mime
View raw message