spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Rods <juanpa.su...@gmail.com>
Subject spark submit with logs and kerberos
Date Mon, 03 Jul 2017 16:19:48 GMT
Hi!

I have a question about logs and have not seen the answer through internet.
I have a spark submit process and I configure a custom log configuration to
it using the next params:
--conf
"spark.executor.extraJavaOptions=-Dlog4j.configuration=customlog4j.properties"
--driver-java-options '-Dlog4j.configuration=customlog4j.properties'
So far is working great, but now I have to add kerberos params like this:
--conf spark.hadoop.fs.hdfs.impl.disable.cache=true
--keytab <keytab path>
--principal <user@domain>
When I add those params (and in particular, --principal) my custom log file
starts being ignored and the default spark log file is being used insted.
Change the default spark configuration file or path is a little difficult
for us since we can't access that cluster.
Is there any way to keep using my custom log configuration the same way? I
am doing something wrong?

Thanks for the help!

Mime
View raw message