spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to Disable or do minimal Logging for apache spark client Driver program?
Date Fri, 07 Oct 2016 05:30:53 GMT
Hi Jakob,
It is a biggest question for me too since I seem to be on a different page than
everyone else whenever I say  "I am also using spark standalonemode and I don't
submit jobs through command line. I just invokepublic static void main() of my
driver program"
Everyone keeps talking about submit jobs from command line or even the words
"submit job" people automatically assume it is happening from command line. I
just setup a standalone cluster and do this




                          SparkConf sparkConf = config.buildSparkConfig();        sparkConf.setJars(JavaSparkContext.jarOfClass(SparkDriver.class));
       JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, new Duration(config.getSparkStremingBatchInterval()));
       ssc.sparkContext().setLogLevel("ERROR");        Receiver receiver = new Receiver(config);
       JavaReceiverInputDStream<String> jsonMessagesDStream = ssc.receiverStream(receiver);
       jsonMessagesDStream.count()        ssc.start();        ssc.awaitTermination();
                

Not using Mixmax yet?
 



so I assume submitting Job happens through this API. please correct me if I am
wrong.
Thanks



On Thu, Oct 6, 2016 1:38 PM, Jakob Odersky jakob@odersky.com
wrote:
You can change the kind of log messages that are shown by

calling "context.setLogLevel(<level>)" with an appropriate level:

ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN.

See
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext@setLogLevel(logLevel:String):Unit

for further details.




Just one nitpick: when you say "I am also using spark standalone

mode and I don't submit jobs through command line. I just invoke

public static void main() of my driver program." are you

referring to spark local mode? It is possible to also run spark

applications in "distributed mode" (i.e. standalone, yarn or

mesos) just from the command line, however that will require

using spark's launcher interface and bundling your application in

a jar.




On Thu, Oct 6, 2016 at 9:27 AM, kant kodali <kanth909@gmail.com> wrote:

> How to Disable or do minimal Logging for apache spark client Driver program?

> I couldn't find this information on docs. By Driver program I mean the java

> program where I initialize spark context. It produces lot of INFO messages

> but I would like to know only when there is error or a Exception such as

> Nullpointer exception and so on. I am also using spark standalone mode and I

> don't submit jobs through command line. I just invoke public static void

> main() of my driver program.
Mime
View raw message