spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From geoHeil <>
Subject Dynamic spark sql
Date Tue, 13 Dec 2016 06:23:39 GMT

I am curious how to dynamically generate spark sql in the scala api.

>From this list val columnsFactor = Seq("bar", "baz")
I want to generate multiple withColumn statements 

dfWithNewLabels.withColumn("replace", lit(null: String))
        .withColumn("bar", when('bar isin (fittedLabels(0).labels: _*),
        .withColumn("baz", when('baz isin (fittedLabels(1).labels: _*),

Kind regards 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:

View raw message