I am using Spark on Kubernetes from Spark 2.4.3. I have created a log4j.properties file in my local spark/conf directory and modified it so that the console (or, in the case of Kubernetes, the log) only shows warnings and higher (log4j.rootCategory=WARN, console). I then added the command

COPY conf /opt/spark/conf

to /root/spark/kubernetes/dockerfiles/spark/Dockerfile and built a new container.

 

However, when I run that under Kubernetes, the program runs successfully but /opt/spark/conf/log4j.properties is not used (I still see the INFO lines when I run kubectl logs <driver pod>).

 

I have tried other things such as explicitly adding a –properties-file to my spark-submit command and even

--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/spark/conf/log4j.properties

 

My log4j.properties file is never seen.

 

How do I customize log4j.properties with Kubernetes?

 

Thanks, Dave Jaffe