spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From German Schiavon <gschiavonsp...@gmail.com>
Subject Re: Spark DF does not rename the column
Date Mon, 04 Jan 2021 19:49:34 GMT
Hi,

I think you have a typo :

root

 |-- ceated: string (nullable = true)


and then:

    withColumnRenamed("created","Date Calculated").


On Mon, 4 Jan 2021 at 19:12, Lalwani, Jayesh <jlalwani@amazon.com.invalid>
wrote:

> You don’t have a column named “created”. The column name is “ceated”,
> without the “r”
>
>
>
> *From: *Mich Talebzadeh <mich.talebzadeh@gmail.com>
> *Date: *Monday, January 4, 2021 at 1:06 PM
> *To: *"user @spark" <user@spark.apache.org>
> *Subject: *[EXTERNAL] Spark DF does not rename the column
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi,
>
>
>
>  version 2.4.3
>
>
>
> I don't know the cause of this.
>
>
>
> This renaming of DF columns used to work fine. I did couple of changes to
> spark/Scala code not relevant to this table and it refuses to rename the
> columns for a table!.
>
>
>
> val summaryACC = HiveContext.table("summaryACC")
>
>
>
> summaryACC.printSchema()
>
>
>
> root
>
>  |-- ceated: string (nullable = true)
>
>  |-- hashtag: string (nullable = true)
>
>  |-- paid: float (nullable = true)
>
>  |-- received: float (nullable = true)
>
>
>
> summaryACC.
>
>     orderBy(desc("paid"),desc("received")).
>
>     withColumnRenamed("created","Date Calculated").
>
>     withColumnRenamed("hashtag","Who").
>
>     withColumn(("received"),format_number(col("received"),2)).
>
>     withColumn(("paid"),format_number(col("paid"),2)).
>
>     withColumnRenamed("paid","paid out/GBP").
>
>     withColumnRenamed("received","paid in/GBP").
>
>     withColumn("paid in/GBP",when(col("paid in/GBP") ===
> "0.00","--").otherwise(col("paid in/GBP"))).
>
>     withColumn("paid out/GBP",when(col("paid out/GBP") ===
> "0.00","--").otherwise(col("paid out/GBP"))).
>
>     select("Date Calculated","Who","paid in/GBP","paid
> out/GBP").show(1000,false)
>
>
>
> and this is the error
>
>
>
> org.apache.spark.sql.AnalysisException: cannot resolve '`Date Calculated`'
> given input columns: [alayer.summaryacc.ceated, Who, paid out/GBP, paid
> in/GBP];;
>
>
>
> This used to work before!
>
>
>
> +----------------------------+------------------+-----------+------------+
>
> |Date Calculated             |Who               |paid in/GBP|paid out/GBP|
>
> +----------------------------+------------------+-----------+------------+
>
> |Mon Jan 04 14:22:17 GMT 2021|paypal            |579.98     |1,526.86    |
>
>
>
> Appreciate any ideas.
>
>
>
> Thanks, Mich
>
>
>

Mime
View raw message