Hi Deepak,

 

The spark logging can be set for different purposes. Say for example if you want to control the spark-submit log, “log4j.logger.org.apache.spark.repl.Main=WARN/INFO/ERROR” can be set.

 

Similarly, to control third party logs:

log4j.logger.org.spark_project.jetty=<LEVEL>, log4j.logger.org.apache.parquet=<LEVEL> etc..

 

These properties can be set in the conf/log4j .properties file.

 

Hope this helps! 😊

 

Regards,

Abhishek Jain

 

From: Deepak Sharma <deepakmca05@gmail.com>
Sent: Thursday, February 14, 2019 12:10 PM
To: spark users <user@spark.apache.org>
Subject: Spark streaming filling the disk with logs

 

Hi All

I am running a spark streaming job with below configuration :

 

--conf "spark.executor.extraJavaOptions=-Droot.logger=WARN,console"

 

But it’s still filling the disk with info logs.

If the logging level is set to WARN at cluster level , then only the WARN logs are getting written but then it affects all the jobs .

 

Is there any way to get rid of INFO level of logging at spark streaming job level ?

 

Thanks

Deepak 

 

--

Thanks
Deepak
www.bigdatabig.com
www.keosha.net