spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Menglong TAN (JIRA)" <>
Subject [jira] [Created] (SPARK-19781) Bucketizer's handleInvalid leave null values untouched unlike the NaNs
Date Wed, 01 Mar 2017 09:35:45 GMT
Menglong TAN created SPARK-19781:

             Summary: Bucketizer's handleInvalid leave null values untouched unlike the NaNs
                 Key: SPARK-19781
             Project: Spark
          Issue Type: Improvement
          Components: MLlib
    Affects Versions: 2.1.0
            Reporter: Menglong TAN
            Priority: Minor

Bucketizer can put NaN values into a special bucket when handleInvalid is on. but leave null
values untouched.

   val data = sc.parallelize(Seq(("crackcell", null.asInstanceOf[java.lang.Double]))).toDF("name",
   val bucketizer = new Bucketizer().setInputCol("number").setOutputCol("number_output").setSplits(Array(Double.NegativeInfinity,
0, 10, Double.PositiveInfinity)).setHandleInvalid("keep")
   val res = bucketizer.transform(data)

will output:

   |     name|number|number_output|
   |crackcell|  null|         null|

If we change null to NaN:

   val data2 = sc.parallelize(Seq(("crackcell", Double.NaN))).toDF("name", "number")
data2: org.apache.spark.sql.DataFrame = [name: string, number: double]

will output:

   |     name|number|number_output|
   |crackcell|   NaN|          3.0|

Maybe we should unify the behaviours? Is it resonable to process nulls as well? If so, maybe
my code can help. :-)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message