spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Jaffe <dja...@vmware.com.INVALID>
Subject Spark on Kubernetes - log4j.properties not read
Date Tue, 11 Jun 2019 01:15:14 GMT
I am using Spark on Kubernetes from Spark 2.4.3. I have created a log4j.properties file in
my local spark/conf directory and modified it so that the console (or, in the case of Kubernetes,
the log) only shows warnings and higher (log4j.rootCategory=WARN, console). I then added the
command
COPY conf /opt/spark/conf
to /root/spark/kubernetes/dockerfiles/spark/Dockerfile and built a new container.

However, when I run that under Kubernetes, the program runs successfully but /opt/spark/conf/log4j.properties
is not used (I still see the INFO lines when I run kubectl logs <driver pod>).

I have tried other things such as explicitly adding a –properties-file to my spark-submit
command and even
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/spark/conf/log4j.properties

My log4j.properties file is never seen.

How do I customize log4j.properties with Kubernetes?

Thanks, Dave Jaffe

Mime
View raw message