spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luciano Resende <luckbr1...@gmail.com>
Subject Re: Create a Column expression from a String
Date Sun, 20 Nov 2016 03:11:06 GMT
Are you looking for UDFs?

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-udfs.html

On Sun, Nov 20, 2016 at 3:12 AM Stuart White <stuart.white1@gmail.com>
wrote:

> I'd like to allow for runtime-configured Column expressions in my
> Spark SQL application.  For example, if my application needs a 5-digit
> zip code, but the file I'm processing contains a 9-digit zip code, I'd
> like to be able to configure my application with the expression
> "substring('zipCode, 0, 5)" to use for the zip code.
>
> So, I think I'm looking for something like this:
>
> def parseColumnExpression(colExpr: String) : Column
>
> I see that SparkSession's sql() method exists to take a string and
> parse it into a DataFrame.  But that's not quite what I want.
>
> Does a mechanism exist that would allow me to take a string
> representation of a column expression and parse it into an actual
> column expression (something that could be use in a .select() call,
> for example)?
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
> --
Sent from my Mobile device

Mime
View raw message