spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From resonance <>
Subject Spark 1.4.1 py4j.Py4JException: Method read([]) does not exist
Date Wed, 12 Aug 2015 12:45:48 GMT
Hello everybody,

I am programming with Pyspark in the Eclipse IDE and have been trying to
transition to Spark 1.4.1 so that I may finally program using Python 3.
(Good work with the progress!) The following program works in Spark 1.3.1
but throws an exception in Spark 1.4.1:

"from pyspark import SparkContext, SparkConf
from pyspark.sql.types import *
from pyspark.sql import SQLContext

if __name__ == '__main__':
    conf = SparkConf().setAppName("MyApp").setMaster("local")
    global sc
    sc = SparkContext(conf=conf)    
    global sqlc
    sqlc = SQLContext(sc)
    symbolsPath = 'SP500Industry.json'
    symbolsRDD = 
    print "Done""

The traceback I'm getting is as follows:

"Traceback (most recent call last):
  File "/media/gavin/20A6-76BF/Current Projects Luna/PySpark Test/",
line 21, in <module>
    symbolsRDD = #rdd with all symbols (and
their industries
"/home/gavin/spark-1.4.1-bin-hadoop2.6/python/pyspark/sql/", line
582, in read
    return DataFrameReader(self)
line 39, in __init__
    self._jreader =
line 538, in __call__
line 304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling Trace:
py4j.Py4JException: Method read([]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.Gateway.invoke(
	at py4j.commands.AbstractCommand.invokeMethod(
	at py4j.commands.CallCommand.execute(

The external libraries I have for the project are
... spark-1.4.1-bin-hadoop2.6/python
... spark-1.4.1-bin-hadoop2.6/python/lib/
... spark-1.4.1-bin-hadoop2.6/python/lib/ (tried both including
and not including this)

Can anybody help me out with what I'm doing wrong?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message