spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Running a spark-submit compatible app in spark-shell
Date Mon, 26 May 2014 21:22:36 GMT
Hi Roger,

This was due to a bug in the Spark shell code, and is fixed in the latest
master (and RC11). Here is the commit that fixed it:
https://github.com/apache/spark/commit/8edbee7d1b4afc192d97ba192a5526affc464205.
Try it now and it should work. :)

Andrew


2014-05-26 10:35 GMT+02:00 Perttu Ranta-aho <rantaaho@iki.fi>:

> Hi Roger,
>
> Were you able to solve this?
>
> -Perttu
>
>
> On Tue, Apr 29, 2014 at 8:11 AM, Roger Hoover <roger.hoover@gmail.com>wrote:
>
>> Patrick,
>>
>> Thank you for replying.  That didn't seem to work either.  I see the
>> option parsed using verbose mode.
>>
>> Parsed arguments:
>>  ...
>>   driverExtraClassPath
>>  /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar
>>
>> But the jar still doesn't show up if I run ":cp" in the repl and the
>> import still fails.
>>
>> scala> import etl._
>> <console>:7: error: not found: value etl
>>        import etl._
>>
>> Not sure if this helps, but I noticed with Spark 0.9.1 that the import
>> only seems to work went I add the -usejavacp option to the spark-shell
>> command.  I don't really understand why.
>>
>> With the latest code, I tried adding these options to the spark-shell
>> command without success: -usejavacp -Dscala.usejavacp=true
>>
>>
>> On Mon, Apr 28, 2014 at 6:30 PM, Patrick Wendell <pwendell@gmail.com>wrote:
>>
>>> What about if you run ./bin/spark-shell
>>> --driver-class-path=/path/to/your/jar.jar
>>>
>>> I think either this or the --jars flag should work, but it's possible
>>> there is a bug with the --jars flag when calling the Repl.
>>>
>>>
>>> On Mon, Apr 28, 2014 at 4:30 PM, Roger Hoover <roger.hoover@gmail.com>wrote:
>>>
>>>> A couple of issues:
>>>> 1) the jar doesn't show up on the classpath even though SparkSubmit had
>>>> it in the --jars options.  I tested this by running > :cp in spark-shell
>>>> 2) After adding it the classpath using (:cp
>>>> /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar),
it
>>>> still fails.  When I do that in the scala repl, it works.
>>>>
>>>> BTW, I'm using the latest code from the master branch
>>>> (8421034e793c0960373a0a1d694ce334ad36e747)
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 3:40 PM, Roger Hoover <roger.hoover@gmail.com>wrote:
>>>>
>>>>> Matei,  thank you.  That seemed to work but I'm not able to import a
>>>>> class from my jar.
>>>>>
>>>>> Using the verbose options, I can see that my jar should be included
>>>>>
>>>>> Parsed arguments:
>>>>> ...
>>>>>   jars
>>>>>  /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar
>>>>>
>>>>> And I see the class I want to load in the jar:
>>>>>
>>>>> jar -tf
>>>>> /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar
|
>>>>> grep IP2IncomeJob
>>>>> etl/IP2IncomeJob$$anonfun$1.class
>>>>> etl/IP2IncomeJob$$anonfun$4.class
>>>>> etl/IP2IncomeJob$.class
>>>>> etl/IP2IncomeJob$$anonfun$splitOverlappingRange$1.class
>>>>> etl/IP2IncomeJob.class
>>>>> etl/IP2IncomeJob$$anonfun$3.class
>>>>> etl/IP2IncomeJob$$anonfun$2.class
>>>>>
>>>>> But the import fails
>>>>>
>>>>> scala> import etl.IP2IncomeJob
>>>>> <console>:10: error: not found: value etl
>>>>>        import etl.IP2IncomeJob
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Apr 27, 2014 at 3:46 PM, Matei Zaharia <
>>>>> matei.zaharia@gmail.com> wrote:
>>>>>
>>>>>> Hi Roger,
>>>>>>
>>>>>> You should be able to use the --jars argument of spark-shell to add
>>>>>> JARs onto the classpath and then work with those classes in the shell.
(A
>>>>>> recent patch, https://github.com/apache/spark/pull/542, made
>>>>>> spark-shell use the same command-line arguments as spark-submit).
But this
>>>>>> is a great question, we should test it out and see whether anything
else
>>>>>> would make development easier.
>>>>>>
>>>>>> SBT also has an interactive shell where you can run classes in your
>>>>>> project, but unfortunately Spark can’t deal with closures typed
directly in
>>>>>> that the right way. However you write your Spark logic in a method
and just
>>>>>> call that method from the SBT shell, that should work.
>>>>>>
>>>>>> Matei
>>>>>>
>>>>>> On Apr 27, 2014, at 3:14 PM, Roger Hoover <roger.hoover@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > From the meetup talk about the 1.0 release, I saw that spark-submit
>>>>>> will be the preferred way to launch apps going forward.
>>>>>> >
>>>>>> > How do you recommend launching such jobs in a development cycle?
>>>>>>  For example, how can I load an app that's expecting to a given to
>>>>>> spark-submit into spark-shell?
>>>>>> >
>>>>>> > Also, can anyone recommend other tricks for rapid development?
 I'm
>>>>>> new to Scala, sbt, etc.  I think sbt can watch for changes in source
files
>>>>>> and compile them automatically.
>>>>>> >
>>>>>> > I want to be able to make code changes and quickly get into
a
>>>>>> spark-shell to play around with them.
>>>>>> >
>>>>>> > I appreciate any advice.  Thanks,
>>>>>> >
>>>>>> > Roger
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message