spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Evan Zamir (JIRA)" <>
Subject [jira] [Created] (SPARK-17508) Setting weightCol to None in ML library causes an error
Date Mon, 12 Sep 2016 16:58:20 GMT
Evan Zamir created SPARK-17508:

             Summary: Setting weightCol to None in ML library causes an error
                 Key: SPARK-17508
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.0.0
            Reporter: Evan Zamir

The following code runs without error:

spark = SparkSession.builder.appName('WeightBug').getOrCreate()
df = spark.createDataFrame(
        (1.0, 1.0, Vectors.dense(1.0)),
        (0.0, 1.0, Vectors.dense(-1.0))
    ["label", "weight", "features"])
lr = LogisticRegression(maxIter=5, regParam=0.0, weightCol="weight")
model =

My expectation from reading the documentation is that setting weightCol=None should treat
all weights as 1.0 (regardless of whether a column exists). However, the same code with weightCol
set to None causes the following errors:

Traceback (most recent call last):

  File "/Users/evanzamir/ams/px-seed-model/scripts/", line 32, in <module>
    model =
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/pyspark/ml/", line 64, in fit
    return self._fit(dataset)
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/pyspark/ml/", line 213, in _fit
    java_model = self._fit_java(dataset)
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/pyspark/ml/", line 210, in _fit_java
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/lib/",
line 933, in __call__
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/pyspark/sql/", line 63, in deco
    return f(*a, **kw)
  File "/usr/local/spark-2.0.0-bin-hadoop2.7/python/lib/",
line 312, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
: java.lang.NullPointerException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at py4j.reflection.MethodInvoker.invoke(
	at py4j.reflection.ReflectionEngine.invoke(
	at py4j.Gateway.invoke(
	at py4j.commands.AbstractCommand.invokeMethod(
	at py4j.commands.CallCommand.execute(

Process finished with exit code 1

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message