spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nan Zhu <zhunanmcg...@gmail.com>
Subject Re: debug standalone Spark jobs?
Date Sun, 05 Jan 2014 18:18:58 GMT
Yes, but my problem only appears when on a large dataset, anyway, Thanks for the reply  

Best,  

--  
Nan Zhu


On Sunday, January 5, 2014 at 11:09 AM, Archit Thakur wrote:

> You can run your spark application locally by setting SPARK_MASTER="local" and then debug
the launched jvm in your IDE.
>  
>  
> On Sun, Jan 5, 2014 at 9:04 PM, Nan Zhu <zhunanmcgill@gmail.com (mailto:zhunanmcgill@gmail.com)>
wrote:
> > Ah, yes, I think application logs really help  
> >  
> > Thank you  
> >  
> > --  
> > Nan Zhu
> >  
> >  
> > On Sunday, January 5, 2014 at 10:13 AM, Sriram Ramachandrasekaran wrote:
> >  
> > > Did you get to look at the spark worker logs? They would be at SPARK_HOME/logs/
> > > Also, you should look at the application logs itself. They would be under SPARK_HOME/work/APP_ID
> > >  
> > >  
> > >  
> > > On Sun, Jan 5, 2014 at 8:36 PM, Nan Zhu <zhunanmcgill@gmail.com (mailto:zhunanmcgill@gmail.com)>
wrote:
> > > > Hi, all  
> > > >  
> > > > I’m trying to run a standalone job in a Spark cluster on EC2,  
> > > >  
> > > > obviously there is some bug in my code, after the job runs for several
minutes, it failed with an exception   
> > > >  
> > > >  
> > > > Loading /usr/share/sbt/bin/sbt-launch-lib.bash
> > > >  
> > > >  
> > > > [info] Set current project to rec_system (in build file:/home/ubuntu/rec_sys/)
> > > >  
> > > >  
> > > > [info] Running general.NetflixRecommender algorithm.SparkALS -b 20 -i
20 -l 0.005 -m spark://172.31.32.76:7077 (http://172.31.32.76:7077) --moviepath s3n://trainingset/netflix/training_set/*
-o s3n://training_set/netflix/training_set/output.txt --rank 20 -r s3n://trainingset/netflix/training_set/mv_*
> > > >  
> > > >  
> > > > log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jEventHandler).
> > > >  
> > > >  
> > > > log4j:WARN Please initialize the log4j system properly.
> > > >  
> > > >  
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
> > > >  
> > > >  
> > > > failed to init the engine class
> > > >  
> > > >  
> > > > org.apache.spark.SparkException: Job aborted: Task 43.0:9 failed more
than 4 times
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:827)
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:825)
> > > >  
> > > >  
> > > > at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
> > > >  
> > > >  
> > > > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:825)
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:440)
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler.org (http://org.apache.spark.scheduler.DAGScheduler.org)$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:502)
> > > >  
> > > >  
> > > > at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:157)
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > >  
> > > > However, this information does not mean anything to me, how can I print
out the detailed log information in console
> > > >  
> > > > I’m not sure about the reasons of those WARNs from log4j, I received
the same WARNING when I run spark-shell, while in there, I can see detailed information like
which task is running, etc.  
> > > >  
> > > > Best,
> > > >  
> > > > --  
> > > > Nan Zhu
> > > >  
> > >  
> > >  
> > >  
> > > --  
> > > It's just about how deep your longing is!
> >  
>  


Mime
View raw message