spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From geoHeil <georg.kf.hei...@gmail.com>
Subject Dynamic spark sql
Date Tue, 13 Dec 2016 06:23:39 GMT
Hi

I am curious how to dynamically generate spark sql in the scala api. 
http://stackoverflow.com/q/41102347/2587904

>From this list val columnsFactor = Seq("bar", "baz")
I want to generate multiple withColumn statements 

dfWithNewLabels.withColumn("replace", lit(null: String))
        .withColumn("bar", when('bar isin (fittedLabels(0).labels: _*),
'baz).otherwise('replace))
        .withColumn("baz", when('baz isin (fittedLabels(1).labels: _*),
'baz).otherwise('replace))
        .drop("replace").show

Kind regards 
Georg



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dynamic-spark-sql-tp28197.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message