spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Friedman <eric.d.fried...@gmail.com>
Subject Re: SparkContext with error from PySpark
Date Tue, 30 Dec 2014 12:57:45 GMT
The Python installed in your cluster is 2.5. You need at least 2.6. 

----
Eric Friedman

> On Dec 30, 2014, at 7:45 AM, Jaggu <jaganadhg@gmail.com> wrote:
> 
> Hi Team,
> 
> I was trying to execute a Pyspark code in cluster. It gives me the following
> error. (Wne I run the same job in local it is working fine too :-()
> 
> Eoor
> 
> Error from python worker:
>  /usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/context.py:209: Warning:
> 'with' will become a reserved keyword in Python 2.6
>  Traceback (most recent call last):
>    File
> "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/runpy.py",
> line 85, in run_module
>      loader = get_loader(mod_name)
>    File
> "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py",
> line 456, in get_loader
>      return find_loader(fullname)
>    File
> "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py",
> line 466, in find_loader
>      for importer in iter_importers(fullname):
>    File
> "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py",
> line 422, in iter_importers
>      __import__(pkg)
>    File "/usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/__init__.py",
> line 41, in <module>
>      from pyspark.context import SparkContext
>    File "/usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/context.py",
> line 209
>      with SparkContext._lock:
>                      ^
>  SyntaxError: invalid syntax
> PYTHONPATH was:
> 
> /usr/lib/spark-1.2.0-bin-hadoop2.3/python:/usr/lib/spark-1.2.0-bin-hadoop2.3/python/lib/py4j-0.8.2.1-src.zip:/usr/lib/spark-1.2.0-bin-hadoop2.3/lib/spark-assembly-1.2.0-hadoop2.3.0.jar:/usr/lib/spark-1.2.0-bin-hadoop2.3/sbin/../python/lib/py4j-0.8.2.1-src.zip:/usr/lib/spark-1.2.0-bin-hadoop2.3/sbin/../python:/home/beehive/bin/utils/primitives:/home/beehive/bin/utils/pylogger:/home/beehive/bin/utils/asterScript:/home/beehive/bin/lib:/home/beehive/bin/utils/init:/home/beehive/installer/packages:/home/beehive/ncli
> java.io.EOFException
>        at java.io.DataInputStream.readInt(DataInputStream.java:392)
>        at
> org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163)
>        at
> org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86)
>        at
> org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62)
>        at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:102)
>        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
>        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
>        at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
>        at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>        at org.apache.spark.scheduler.Task.run(Task.scala:56)
>        at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>        at java.lang.Thread.run(Thread.java:722)
> 
> 14/12/31 04:49:58 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
> 1, aster4, NODE_LOCAL, 1321 bytes)
> 14/12/31 04:49:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
> on aster4:43309 (size: 3.8 KB, free: 265.0 MB)
> 14/12/31 04:49:59 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) on
> executor aster4: org.apache.spark.SparkException (
> 
> 
> Any clue how to resolve the same.
> 
> Best regards
> 
> Jagan
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-with-error-from-PySpark-tp20907.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message