spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
Subject Re: Spark - Not contains on Spark dataframe
Date Sat, 04 Mar 2017 15:45:55 GMT
Hi,

I was able to resolve issue with below conditions.

datapoint_df(Constants.Datapoint.Vin).like("012345")

datapoint_filter_df.filter( datapoint_filter_df(Constants.Datapoint.Vin)
rlike "^([A-Z]|[0-9]|[a-z])+$" ) // for checking alpha numeric.

Thanks,
Asmath

On Tue, Feb 28, 2017 at 10:49 AM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:

> Hi,
>
> Could anyone please provide me your suggestions on how to resolve the
> issue that I am facing with not contains code on dataframe column.
>
> Here is the code. My dataframe is not getting filtered with below
> conditions. I even tried not and ! on Column. any suggestions?
>
> def filterDatapointRawCountsDF(vin: Column): Column =
>
> {
>
> import org.apache.spark.sql.functions.not
>  val filterColumn: Column = {
>
> not(vin.contains("VIN")) ||
> not(vin.contains("Ÿ")) ||
> not(vin.contains("0123456789ABCDEFG"))
>
> }
>
> filterColumn;
>
> }
>
>
> }
>
>

Mime
View raw message