spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sriram Ramachandrasekaran <sri.ram...@gmail.com>
Subject Re: debug standalone Spark jobs?
Date Sun, 05 Jan 2014 15:13:12 GMT
Did you get to look at the spark worker logs? They would be at
SPARK_HOME/logs/
Also, you should look at the application logs itself. They would be under
SPARK_HOME/work/APP_ID



On Sun, Jan 5, 2014 at 8:36 PM, Nan Zhu <zhunanmcgill@gmail.com> wrote:

> Hi, all
>
> I’m trying to run a standalone job in a Spark cluster on EC2,
>
> obviously there is some bug in my code, after the job runs for several
> minutes, it failed with an exception
>
> Loading /usr/share/sbt/bin/sbt-launch-lib.bash
>
> [info] Set current project to rec_system (in build
> file:/home/ubuntu/rec_sys/)
>
> [info] Running general.NetflixRecommender algorithm.SparkALS -b 20 -i 20
> -l 0.005 -m spark://172.31.32.76:7077 --moviepath
> s3n://trainingset/netflix/training_set/* -o
> s3n://training_set/netflix/training_set/output.txt --rank 20 -r
> s3n://trainingset/netflix/training_set/mv_*
>
> log4j:WARN No appenders could be found for logger
> (akka.event.slf4j.Slf4jEventHandler).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> failed to init the engine class
>
> org.apache.spark.SparkException: Job aborted: Task 43.0:9 failed more than
> 4 times
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:827)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:825)
>
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
>
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:825)
>
> at
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:440)
>
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:502)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:157)
>
>
>
>
>
> However, this information does not mean anything to me, how can I print
> out the detailed log information in console
>
> I’m not sure about the reasons of those WARNs from log4j, I received the
> same WARNING when I run spark-shell, while in there, I can see detailed
> information like which task is running, etc.
>
> Best,
>
> --
> Nan Zhu
>
>


-- 
It's just about how deep your longing is!

Mime
View raw message