spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naveen Madhire <vmadh...@umail.iu.edu>
Subject pyspark issue
Date Mon, 27 Jul 2015 21:40:46 GMT
Hi,

I am running pyspark in windows and I am seeing an error while adding
pyfiles to the sparkcontext. below is the example,

sc = SparkContext("local","Sample",pyFiles="C:/sample/yattag.zip")

this fails with no file found error for "C"


The below logic is treating the path as individual files like "C", ":" "/" etc.

https://github.com/apache/spark/blob/master/python/pyspark/context.py#l195


It works if I use Spark Conf,

sparkConf.set("spark.submit.pyFiles",*"**C:/sample/yattag.zip**"*)
sc = SparkContext("local","Sample",conf=sparkConf)


Is this an existing issue or I am not including the files in correct
way in Spark Context?


Thanks.





when I run this, I am getting

Mime
View raw message