spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eric wong <win19...@gmail.com>
Subject Re: how to set log level of spark executor on YARN(using yarn-cluster mode)
Date Mon, 20 Oct 2014 07:22:23 GMT
Thanks for your reply!

Sorry for i forgot referring the spark which i'm using is *Version 1.0.2*
instead of 1.1.0.

Also the document of 1.0.2 seems not same like 1.1.0:
http://spark.apache.org/docs/1.0.2/running-on-yarn.html

And i tried your suggestion(upload ) but did not work:

*1. set my copy of log4j.properties like:*

*log4j.rootCategory=DEBUG, console*


*2. upload when using spark-submit script:*

*./bin/spark-submit --class edu.bjut.spark.SparkPageRank     --master
yarn-cluster     --num-executors 5     --executor-memory 2g
--executor-cores 1     /data/hadoopspark/MySparkTest.jar
hdfs://master:8000/srcdata/searchengine/* 5 5
hdfs://master:8000/resultdata/searchengine/2014102001/ *
*--files log4j.properties*

So plz point out my fault and any suggestion would be welcome

Thanks!



2014-10-16 9:45 GMT+08:00 Marcelo Vanzin <vanzin@cloudera.com>:

> Hi Eric,
>
> Check the "Debugging Your Application" section at:
> http://spark.apache.org/docs/latest/running-on-yarn.html
>
> Long story short: upload your log4j.properties using the "--files"
> argument of spark-submit.
>
> (Mental note: we could make the log level configurable via a system
> property...)
>
>
> On Wed, Oct 15, 2014 at 5:58 PM, eric wong <win19999@gmail.com> wrote:
> > Hi,
> >
> > I want to check the DEBUG log of spark executor on YARN(using
> yarn-cluster
> > mode), but
> >
> > 1. yarn daemonlog setlevel DEBUG YarnChild.class
> > 2. set log4j.properties in spark/conf folder on client node.
> >
> > no means above works.
> >
> > So how could i set the  log level of spark executor on YARN container to
> > DEBUG?
> >
> > Thanks!
> >
> >
> >
> >
> > --
> > Wang Haihua
> >
>
>
>
> --
> Marcelo
>



-- 
王海华

Mime
View raw message