spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bedrytski Aliaksandr <sp...@bedryt.ski>
Subject Re: how to find NaN values of each row of spark dataframe to decide whether the rows is dropeed or not
Date Mon, 26 Sep 2016 07:53:40 GMT
Hi Muhammet,

have you tried to use sql queries?

> spark.sql("""
>     SELECT
>         field1,
>         field2,
>         field3
>    FROM table1
>    WHERE
>         field1 != 'Nan',
>         field2 != 'Nan',
>         field3 != 'Nan'
> """)

This query filters rows containing Nan for a table with 3 columns.

Regards,
--
  Bedrytski Aliaksandr
  spark@bedryt.ski



On Mon, Sep 26, 2016, at 09:30, muhammet paky├╝rek wrote:
>
> is there any way to do this directly.  if its not, is there any todo
> this indirectly using another datastrcutures of spark
>

Mime
View raw message