spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Dynamically get value based on Map key in Spark Dataframe
Date Mon, 18 Jul 2016 15:06:42 GMT
See broadcast variable.

Or (just a thought) do join between DataFrames.

Jacek

On 18 Jul 2016 9:24 a.m., "Divya Gehlot" <divya.htconex@gmail.com> wrote:

> Hi,
>
> I have created a map by reading a text file
> val keyValueMap = file_read.map(t => t.getString(0) ->
> t.getString(4)).collect().toMap
>
> Now I have another dataframe where I need to dynamically replace all the
> keys of Map with values
> val df_input = reading the file as dataframe
> val df_replacekeys =
> df_input.withColumn("map_values",lit(keyValueMap (col("key"))))
>
> Would really appreciate the help .
>
>
> Thanks,
> Divya
>
>
>

Mime
View raw message