spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maciej Szymkiewicz <mszymkiew...@gmail.com>
Subject [ML][PYTHON] Collecting data in a class extending SparkSessionTestCase causes AttributeError:
Date Tue, 07 Mar 2017 03:29:53 GMT
Hi everyone,

It is a either to late or to early for me to think straight so please
forgive me if it is something trivial. I am trying to add a test case
extending SparkSessionTestCase to pyspark.ml.tests (example patch
attached). If test collects data, and there is another TestCase
extending extending SparkSessionTestCase executed before it, I get
AttributeError due to _jsc being None:

======================================================================

ERROR: test_foo (pyspark.ml.tests.FooTest)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/spark/python/pyspark/ml/tests.py", line 1258, in test_foo

  File "/home/spark/python/pyspark/sql/dataframe.py", line 389, in collect

    with SCCallSiteSync(self._sc) as css:

  File "/home/spark/python/pyspark/traceback_utils.py", line 72, in __enter__

    self._context._jsc.setCallSite(self._call_site)

AttributeError: 'NoneType' object has no attribute 'setCallSite'

----------------------------------------------------------------------

If TestCase is executed alone it seems to work just fine.


Can anyone reproduce this? Is there something obvious I miss here?

-- 
Best,
Maciej


Mime
View raw message