spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anbutech <anbutec...@outlook.com>
Subject How to handle Null values in Array of struct elements in pyspark
Date Wed, 08 Apr 2020 06:10:18 GMT
Hello All,

We have a data in a column in pyspark dataframe having array of struct type
having multiple nested fields present.if the value is not blank it will save
the data in the same array of struct type in spark delta table.

please advise on the below case:

if the same column coming as blank ,it is treated as array<string> in the
dataframe.

f the same column coming as array of struct elements it is fine.

Could you please advise how do we handle this scenario in pyspark to avoid
the error "Failed to merge incompatible data types StructType(StructField(ac
...................... and StringType

how do we handle  array of struct type and array<string> in the same column

Thanks
Anbu


Thanks
Anbu



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message