spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Maas <gerard.m...@gmail.com>
Subject Spark Job Server first steps
Date Thu, 22 May 2014 17:25:35 GMT
Hi,

I'm starting to explore the Spark Job Server contributed by Ooyala [1],
running from the master branch.

I started by developing and submitting a simple job and the JAR check gave
me errors on a seemingly good jar. I disabled the fingerprint checking on
the jar and  I could submit it, but when I tried to submit the job, it
could not find it's classpath. Therefore I decided to take a couple of
steps backwards and go through the example in the docs.

Using the (Hello)WordCount example, upload is OK and the jar in in the UI
as well, but when I submit the job, I get the same classpathNotFound error
as before:

19:07 $ curl -d ""
'localhost:8090/jobs?appName=test&classPath=spark.jobserver.WordCountExample'
{
  "status": "ERROR",
  "result": "classPath spark.jobserver.WordCountExample not found"
}

I'm not sure where it goes wrong.  Here's what seems to be the relevant
snippet in the server logs:

[2014-05-22 19:17:28,891] INFO  .apache.spark.SparkContext []
[akka://JobServer/user/context-
supervisor/666d021a-spark.jobserver.WordCountExample] - Added JAR
/tmp/spark-jobserver/filedao/data/test-2014-05-22T18:44:09.254+02:00.jar at
http://172.17.42.1:37978/jars/test-2014-05-22T18:44:09.254+02:00.jar with
timestamp 1400779048891
[2014-05-22 19:17:28,891] INFO  util.ContextURLClassLoader []
[akka://JobServer/user/context-supervisor/666d021a-spark.jobserver.WordCountExample]
- Added URL
file:/tmp/spark-jobserver/filedao/data/test-2014-05-22T18:44:09.254+02:00.jar
to ContextURLClassLoader
[2014-05-22 19:17:28,891] INFO  spark.jobserver.JarUtils$ []
[akka://JobServer/user/context-supervisor/666d021a-spark.jobserver.WordCountExample]
- Loading object spark.jobserver.WordCountExample$ using loader
spark.jobserver.util.ContextURLClassLoader@5deae1b7
[2014-05-22 19:17:28,892] INFO  spark.jobserver.JarUtils$ []
[akka://JobServer/user/context-supervisor/666d021a-spark.jobserver.WordCountExample]
- Loading class spark.jobserver.WordCountExample using loader
spark.jobserver.util.ContextURLClassLoader@5deae1b7

***** all OK until here and then ...*****

[2014-05-22 19:17:28,892] INFO  ocalContextSupervisorActor []
[akka://JobServer/user/context-supervisor] - Shutting down context
666d021a-spark.jobserver.WordCountExample

Any ideas? Something silly I might be doing?  btw, I'm running in dev mode
using sbt and default config (local).

-kr, Gerard.


[1] https://github.com/ooyala/spark-jobserver

Mime
View raw message