spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Starks <>
Subject Newbie question on how to extract column value
Date Tue, 07 Aug 2018 15:09:11 GMT
I am very new to Spark. Just successfully setup Spark SQL connecting to postgresql database,
and am able to display table with code

    sparkSession.sql("SELECT id, url from table_a where col_b <> '' ").show()

Now I want to perform filter and map function on col_b value. In plain scala it would be something

    Seq((1, ""), (2, ""), (3, "unknown")).filter { case (_, url)
=> isValid(url) }.map { case (id, url)  => (id, pathOf(url)) }

where filter will remove invalid url, and then map (id, url) to (id, path of url).

However, when applying this concept to spark sql with code snippet


Compiler complains type mismatch because $"url" is ColumnName type. How can I extract column
value i.e. http://... for the column url in order to perform filter function?


Java 1.8.0
Scala 2.11.8
Spark 2.1.0
View raw message