spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Davidson <>
Subject how to set up pyspark eclipse, pyDev, virtualenv? syntaxError: yield from walk(
Date Thu, 05 Apr 2018 00:36:43 GMT
I am having a heck of a time setting up my development environment. I used
pip to install pyspark. I also downloaded spark from apache.

My eclipse pyDev intereperter is configured as a python3 virtualenv

I have a simple unit test that loads a small dataframe. generates
the following error

2018-04-04 17:13:56 ERROR Executor:91 - Exception in task 0.0 in stage 0.0
(TID 0)


Error from python worker:

  Traceback (most recent call last):

    File "/Users/a/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/",
line 67, in <module>

      import os

    File "/Users/a/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/",
line 409

      yield from walk(new_path, topdown, onerror, followlinks)


  SyntaxError: invalid syntax

My unittest classs is dervied from.

class PySparkTestCase(unittest.TestCase):


    def setUpClass(cls):

        conf = SparkConf().setMaster("local[2]") \

            .setAppName(cls.__name__) #\

#             .set("spark.authenticate.secret", "111111")

        cls.sparkContext = SparkContext(conf=conf)

        sc_values[cls.__name__] = cls.sparkContext

        cls.sqlContext = SQLContext(cls.sparkContext)

        print("aedwip:", SparkContext)


    def tearDownClass(cls):

        print("....calling stop tearDownClas, the content of sc_values=",



This looks similar to Class  PySparkTestCase in

Any suggestions would be greatly appreciated.


My downloaed version is spark-2.3.0-bin-hadoop2.7

My virtual env version is

(spark-2.3.0) $ pip show pySpark

Name: pyspark

Version: 2.3.0

Summary: Apache Spark Python API


Author: Spark Developers




Requires: py4j

(spark-2.3.0) $ 

(spark-2.3.0) $ python --version

Python 3.6.1

(spark-2.3.0) $ 

View raw message