spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anbutech <anbutec...@outlook.com>
Subject Explode/Flatten Map type Data Using Pyspark
Date Thu, 14 Nov 2019 17:50:02 GMT
Hello Sir,

I have a scenario to flatten the different combinations of map type(key
value) in a column called eve_data  like below:

How do we flatten the map type into proper columns using pyspark


1) Source Dataframe having 2 columns(event id,data)

eve_id,eve_data
001,  "k1":"abc",
      "k2":"xyz"
	  "k3":"10091"

eve_id,eve_data

002,   "k1":"12",
          "k2":"jack",
	   "k3":"0.01",
	   "k4":"0998"

eve_id,eve_data	

003,   "k1":"aaa",
         "k2":"xxxx",
	  "k3":"device",
	  "k4":"endpoint",
	  "k5":"-"
	
	
Final output:

(flatten the output of each  event ids key values).The number of key values
will be different for each event id.so i want to flatten the records for all
the map type(key values) as below
	
eve_id	k1  k2  k3
001        abc xyz 10091

eve_id,  k1  k2   k3   k4
002,     12  jack 0.01 0998

eve_id,   k1     k2        k3          k4      k5
003,       aaa  xxxx   device endpoint     -


Thanks
Anbu



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message