spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Davidson <A...@SantaCruzIntegration.com>
Subject how to set up pyspark eclipse, pyDev, virtualenv? syntaxError: yield from walk(
Date Thu, 05 Apr 2018 00:36:43 GMT
I am having a heck of a time setting up my development environment. I used
pip to install pyspark. I also downloaded spark from apache.

My eclipse pyDev intereperter is configured as a python3 virtualenv

I have a simple unit test that loads a small dataframe. Df.show() generates
the following error


2018-04-04 17:13:56 ERROR Executor:91 - Exception in task 0.0 in stage 0.0
(TID 0)

org.apache.spark.SparkException:

Error from python worker:

  Traceback (most recent call last):

    File "/Users/a/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site.py",
line 67, in <module>

      import os

    File "/Users/a/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/os.py",
line 409

      yield from walk(new_path, topdown, onerror, followlinks)

               ^

  SyntaxError: invalid syntax





My unittest classs is dervied from.



class PySparkTestCase(unittest.TestCase):



    @classmethod

    def setUpClass(cls):

        conf = SparkConf().setMaster("local[2]") \

            .setAppName(cls.__name__) #\

#             .set("spark.authenticate.secret", "111111")

        cls.sparkContext = SparkContext(conf=conf)

        sc_values[cls.__name__] = cls.sparkContext

        cls.sqlContext = SQLContext(cls.sparkContext)

        print("aedwip:", SparkContext)



    @classmethod

    def tearDownClass(cls):

        print("....calling stop tearDownClas, the content of sc_values=",
sc_values)

        sc_values.clear()

        cls.sparkContext.stop()



This looks similar to Class  PySparkTestCase in
https://github.com/apache/spark/blob/master/python/pyspark/tests.py



Any suggestions would be greatly appreciated.



Andy



My downloaed version is spark-2.3.0-bin-hadoop2.7



My virtual env version is

(spark-2.3.0) $ pip show pySpark

Name: pyspark

Version: 2.3.0

Summary: Apache Spark Python API

Home-page: https://github.com/apache/spark/tree/master/python

Author: Spark Developers

Author-email: dev@spark.apache.org

License: http://www.apache.org/licenses/LICENSE-2.0

Location: 
/Users/a/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages

Requires: py4j

(spark-2.3.0) $ 



(spark-2.3.0) $ python --version

Python 3.6.1

(spark-2.3.0) $ 





Mime
View raw message