spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: 0.9.0 forces log4j usage
Date Fri, 07 Feb 2014 08:05:40 GMT
A config option e.g. could just be to add:

spark.logging.loadDefaultLogger (default true)
If set to true, Spark will try to initialize a log4j logger if none is
detected. Otherwise Spark will not modify logging behavior.

Then users could just set this to false if they have a logging set-up
that conflicts with this.

Maybe there is a nicer fix...

On Fri, Feb 7, 2014 at 12:03 AM, Patrick Wendell <> wrote:
> Hey Paul,
> Thanks for digging this up. I worked on this feature and the intent
> was to give users good default behavior if they didn't include any
> logging configuration on the classpath.
> The problem with assuming that CL tooling is going to fix the job is
> that many people link against spark as a library and run their
> application using their own scripts. In this case the first thing
> people see when they run an application that links against Spark was a
> big ugly logging warning.
> I'm not super familiar with log4j-over-slf4j, but this behavior of
> returning null for the appenders seems a little weird. What is the use
> case for using this and not just directly use slf4j-log4j12 like Spark
> itself does?
> Did you have a more general fix for this in mind? Or was your plan to
> just revert the existing behavior... We might be able to add a
> configuration option to disable this logging default stuff. Or we
> could just rip it out - but I'd like to avoid that if possible.
> - Patrick
> On Thu, Feb 6, 2014 at 11:41 PM, Paul Brown <> wrote:
>> We have a few applications that embed Spark, and in 0.8.0 and 0.8.1, we
>> were able to use slf4j, but 0.9.0 broke that and unintentionally forces
>> direct use of log4j as the logging backend.
>> The issue is here in the org.apache.spark.Logging trait:
>> log4j-over-slf4j *always* returns an empty enumeration for appenders to the
>> ROOT logger:
>> And this causes an infinite loop and an eventual stack overflow.
>> I'm happy to submit a Jira and a patch, but it would be significant enough
>> reversal of recent changes that it's probably worth discussing before I
>> sink a half hour into it.  My suggestion would be that initialization (or
>> not) should be left to the user with reasonable default behavior supplied
>> by the spark commandline tooling and not forced on applications that
>> incorporate Spark.
>> Thoughts/opinions?
>> -- Paul
>> --
>> | Multifarious, Inc. |

View raw message