I am using Spark on Kubernetes from Spark 2.4.3. I have created a log4j.properties file in my local spark/conf directory and modified it so that the console (or, in the case of Kubernetes, the log) only shows warnings and higher (log4j.rootCategory=WARN, console). I then added the command
COPY conf /opt/spark/conf
to /root/spark/kubernetes/dockerfiles/spark/Dockerfile and built a new container.
However, when I run that under Kubernetes, the program runs successfully but /opt/spark/conf/log4j.properties is not used (I still see the INFO lines when I run kubectl logs <driver pod>).
I have tried other things such as explicitly adding a –properties-file to my spark-submit command and even
My log4j.properties file is never seen.
How do I customize log4j.properties with Kubernetes?
Thanks, Dave Jaffe