spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: spark-submit and logging
Date Thu, 20 Nov 2014 11:08:16 GMT
I think the standard practice is to include your log config file among
the files uploaded to YARN containers, and then set
-Dlog4j.configuration=yourfile.xml in
spark.{executor.driver}.extraJavaOptions ?

On Thu, Nov 20, 2014 at 9:20 AM, Tobias Pfeiffer <> wrote:
> Hi,
> I am using spark-submit to submit my application jar to a YARN cluster.  I
> want to deliver a single jar file to my users, so I would like to avoid to
> tell them "also, please put that log4j.xml file somewhere and add that path
> to the spark-submit command".
> I thought it would be sufficient that my application jar file contains a
> log4j.xml file, but that does not seem to be the case.  If I don't add a
> log4j.xml file to the classpath before launching spark-submit, the one
> bundled with spark will be used -- which has a negative influence on my
> program execution.  Is there any way I can tell spark-submit to use the
> log4j configuration bundled in my jar file?
> Thanks
> Tobias

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message