spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillaume Charhon <guilla...@databerries.com>
Subject Re: Error when using multiple python files spark-submit
Date Fri, 20 Mar 2015 08:48:37 GMT
I see. I will try the other way around.

On Thu, Mar 19, 2015 at 8:06 PM, Davies Liu <davies@databricks.com> wrote:

> the options of spark-submit should come before main.py, or they will
> become the options of main.py, so it should be:
>
>      ../hadoop/spark-install/bin/spark-submit --py-files
>
>  /home/poiuytrez/naive.py,/home/poiuytrez/processing.py,/home/poiuytrez/settings.py
>  --master spark://spark-m:7077 main.py
>
> On Mon, Mar 16, 2015 at 4:11 AM, poiuytrez <guillaume@databerries.com>
> wrote:
> > I have a spark app which is composed of multiple files.
> >
> > When I launch Spark using:
> >
> >     ../hadoop/spark-install/bin/spark-submit main.py --py-files
> >
> /home/poiuytrez/naive.py,/home/poiuytrez/processing.py,/home/poiuytrez/settings.py
> > --master spark://spark-m:7077
> >
> > I am getting an error:
> >
> >     15/03/13 15:54:24 INFO TaskSetManager: Lost task 6.3 in stage 413.0
> (TID
> > 5817) on executor spark-w-3.c.databerries.internal:
> > org.apache.spark.api.python.PythonException (Traceback (most recent call
> > last):   File "/home/hadoop/spark-install/python/pyspark/worker.py", line
> > 90, in main
> >         command = pickleSer._read_with_length(infile)   File
> > "/home/hadoop/spark-install/python/pyspark/serializers.py", line 151, in
> > _read_with_length
> >         return self.loads(obj)   File
> > "/home/hadoop/spark-install/python/pyspark/serializers.py", line 396, in
> > loads
> >         return cPickle.loads(obj) ImportError: No module named naive
> >
> > It is weird because I do not serialize anything. naive.py is also
> available
> > on every machine at the same path.
> >
> > Any insight on what could be going on? The issue does not happen on my
> > laptop.
> >
> > PS : I am using Spark 1.2.0.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-using-multiple-python-files-spark-submit-tp22080.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>

Mime
View raw message