spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillaume Charhon <guilla...@databerries.com>
Subject Re: Spark-submit and multiple files
Date Fri, 20 Mar 2015 08:47:04 GMT
Hi Davies,

I am already using --py-files. The system does use the other file. The
error I am getting is not trivial. Please check the error log.



On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <davies@databricks.com> wrote:

> You could submit additional Python source via --py-files , for example:
>
> $ bin/spark-submit --py-files work.py main.py
>
> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <guillaume@databerries.com>
> wrote:
> > Hello guys,
> >
> > I am having a hard time to understand how spark-submit behave with
> multiple
> > files. I have created two code snippets. Each code snippet is composed
> of a
> > main.py and work.py. The code works if I paste work.py then main.py in a
> > pyspark shell. However both snippets do not work when using spark submit
> and
> > generate different errors.
> >
> > Function add_1 definition outside
> > http://www.codeshare.io/4ao8B
> > https://justpaste.it/jzvj
> >
> > Embedded add_1 function definition
> > http://www.codeshare.io/OQJxq
> > https://justpaste.it/jzvn
> >
> > I am trying a way to make it work.
> >
> > Thank you for your support.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>

Mime
View raw message