spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Confused about class paths in spark 1.1.0
Date Fri, 31 Oct 2014 01:15:57 GMT
Yeah, I think you should file this as a bug. The problem is that JARs need to also be added
into the Scala compiler and REPL class loader, and we probably don't do this for the ones
in this driver config property.

Matei

> On Oct 30, 2014, at 6:07 PM, Shay Seng <shay@urbanengines.com> wrote:
> 
> -- jars does indeed work but this causes the "jars" to also get shipped to the workers
-- which I don't want to do for efficiency reasons.
> 
> I think you are saying that setting "spark.driver.extraClassPath" in spark-default.conf
 ought to have the same behavior as providing "--driver.class.apth"  to spark-shell. Correct?
If so I will file a bug report since this is definitely not the case.
> 
> 
> On Thu, Oct 30, 2014 at 5:39 PM, Matei Zaharia <matei.zaharia@gmail.com <mailto:matei.zaharia@gmail.com>>
wrote:
> Try using --jars instead of the driver-only options; they should work with spark-shell
too but they may be less tested.
> 
> Unfortunately, you do have to specify each JAR separately; you can maybe use a shell
script to list a directory and get a big list, or set up a project that builds all of the
dependencies into one assembly JAR.
> 
> Matei
> 
> > On Oct 30, 2014, at 5:24 PM, Shay Seng <shay@urbanengines.com <mailto:shay@urbanengines.com>>
wrote:
> >
> > Hi,
> >
> > I've been trying to move up from spark 0.9.2 to 1.1.0.
> > I'm getting a little confused with the setup for a few different use cases, grateful
for any pointers...
> >
> > (1) spark-shell + with jars that are only required by the driver
> > (1a)
> > I added "spark.driver.extraClassPath  /mypath/to.jar" to my spark-defaults.conf
> > I launched spark-shell with:  ./spark-shell
> >
> > Here I see on the WebUI that spark.driver.extraClassPath has been set, but I am
NOT able to access any methods in the jar.
> >
> > (1b)
> > I removed "spark.driver.extraClassPath" from my spark-default.conf
> > I launched spark-shell with  .//spark-shell --driver.class.path /mypath/to.jar
> >
> > Again I see that the WebUI spark.driver.extraClassPath has been set.
> > But this time I am able to access the methods in the jar.
> >
> > Q: Is spark-shell not considered the driver in this case?  why does using --driver.class.path
on the command line have a different behavior to setting it in spark-defaults.conf ?
> >
> >
> > (2) Rather than adding each jar individually, is there a way to use wildcards? Previously
with SPARK_CLASS_PATH I was able to use <mypath>/*  but with --driver.class.path it
seems to require individual files.
> >
> > tks
> > Shay
> 
> 


Mime
View raw message