spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: Explode/Flatten Map type Data Using Pyspark
Date Fri, 15 Nov 2019 03:16:38 GMT
Hi

How do you want your final DF to look like? Is it with all 5 value columns?
Do you have a finite set of columns?

On Fri, Nov 15, 2019 at 4:50 AM anbutech <anbutech17@outlook.com> wrote:

> Hello Sir,
>
> I have a scenario to flatten the different combinations of map type(key
> value) in a column called eve_data  like below:
>
> How do we flatten the map type into proper columns using pyspark
>
>
> 1) Source Dataframe having 2 columns(event id,data)
>
> eve_id,eve_data
> 001,  "k1":"abc",
>       "k2":"xyz"
>           "k3":"10091"
>
> eve_id,eve_data
>
> 002,   "k1":"12",
>           "k2":"jack",
>            "k3":"0.01",
>            "k4":"0998"
>
> eve_id,eve_data
>
> 003,   "k1":"aaa",
>          "k2":"xxxx",
>           "k3":"device",
>           "k4":"endpoint",
>           "k5":"-"
>
>
> Final output:
>
> (flatten the output of each  event ids key values).The number of key values
> will be different for each event id.so i want to flatten the records for
> all
> the map type(key values) as below
>
> eve_id  k1  k2  k3
> 001        abc xyz 10091
>
> eve_id,  k1  k2   k3   k4
> 002,     12  jack 0.01 0998
>
> eve_id,   k1     k2        k3          k4      k5
> 003,       aaa  xxxx   device endpoint     -
>
>
> Thanks
> Anbu
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

-- 
Best Regards,
Ayan Guha

Mime
View raw message