spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debabrata Ghosh <mailford...@gmail.com>
Subject Need help with String Concat Operation
Date Wed, 18 Oct 2017 06:17:57 GMT
Hi,
             I am having a dataframe column (name of the column is CTOFF)
and I intend to prefix with '0' in case the length of the column is 3.
Unfortunately, I am unable to acheive my goal and wonder whether you can
help me here.

Command which I am executing:

ctoff_dedup_prep_temp =
ctoff_df.withColumn('CTOFF_NEW',when(length(col('CTOFF')) ==
3,'0'+col('CTOFF')))
ctoff_dedup_prep_temp.show()

+------------+----------+----------+--------------------+---------+-----+---------+
|EVNT_SRVC_AR|EVNT_FCLTY|EVNT_TP_CD|        NTWRK_PRD_CD|
DY_OF_WK|CTOFF|CTOFF_NEW|
+------------+----------+----------+--------------------+---------+-----+---------+
|         HKG|       HKC|        AR|2,3,7,8,C,D,J,P,Q...|1,2,3,4,5|
1440|     null|
|         HKG|       HKC|        AR|             C,Q,T,Y|1,2,3,4,5|
730|    730.0|
|         HKG|       HKC|        AR|         E,K,C,Q,T,Y|1,2,3,4,5|
600|    600.0|
|         HKG|       HKC|        AR|         E,K,C,Q,T,Y|1,2,3,4,5|
900|    900.0|
+------------+----------+----------+--------------------+---------+-----+---------+

The result which I want is:
+------------+----------+----------+--------------------+---------+-----+---------+
|EVNT_SRVC_AR|EVNT_FCLTY|EVNT_TP_CD|        NTWRK_PRD_CD|
DY_OF_WK|CTOFF|CTOFF_NEW|
+------------+----------+----------+--------------------+---------+-----+---------+
|         HKG|       HKC|        AR|2,3,7,8,C,D,J,P,Q...|1,2,3,4,5|
1440|    1440|
|         HKG|       HKC|        AR|             C,Q,T,Y|1,2,3,4,5|
730|    0730|
|         HKG|       HKC|        AR|         E,K,C,Q,T,Y|1,2,3,4,5|
600|    0600|
|         HKG|       HKC|        AR|         E,K,C,Q,T,Y|1,2,3,4,5|
900|    0900|
+------------+----------+----------+--------------------+---------+-----+---------+

So I want the '0' to be prefixed but it's getting suffixed as '.0'. Any
clue around why is this happening

Thanks,

Debu

Mime
View raw message