spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: SPARK_SUBMIT_CLASSPATH question
Date Thu, 16 Oct 2014 00:53:00 GMT
Hi Greg,

I'm not sure exactly what it is that you're trying to achieve, but I'm
pretty sure those variables are not supposed to be set by users. You
should take a look at the documentation for
"spark.driver.extraClassPath" and "spark.driver.extraLibraryPath", and
the equivalent options for executors. (The driver ones also have
direct equivalents in the spark-submit command line, check its help
output.)

Since you're running on Yarn, you might also want to take a look at
https://issues.apache.org/jira/browse/SPARK-1719 (and SPARK-1720).

On Tue, Oct 14, 2014 at 11:57 AM, Greg Hill <greg.hill@rackspace.com> wrote:
> It seems to me that SPARK_SUBMIT_CLASSPATH does not follow the same ability
> as other tools to put wildcards in the paths you add.  For some reason it
> doesn't pick up the classpath information from yarn-site.xml either, it
> seems, when running on YARN.  I'm having to manually add every single
> dependency JAR.  There must be a better way, so what am I missing?
>
> Greg
>



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message