spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anbutech <>
Subject Explode/Flatten Map type Data Using Pyspark
Date Thu, 14 Nov 2019 17:50:02 GMT
Hello Sir,

I have a scenario to flatten the different combinations of map type(key
value) in a column called eve_data  like below:

How do we flatten the map type into proper columns using pyspark

1) Source Dataframe having 2 columns(event id,data)

001,  "k1":"abc",


002,   "k1":"12",


003,   "k1":"aaa",
Final output:

(flatten the output of each  event ids key values).The number of key values
will be different for each event i want to flatten the records for all
the map type(key values) as below
eve_id	k1  k2  k3
001        abc xyz 10091

eve_id,  k1  k2   k3   k4
002,     12  jack 0.01 0998

eve_id,   k1     k2        k3          k4      k5
003,       aaa  xxxx   device endpoint     -


Sent from:

To unsubscribe e-mail:

View raw message