spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: 0.9.0 forces log4j usage
Date Fri, 07 Feb 2014 08:03:04 GMT
Hey Paul,

Thanks for digging this up. I worked on this feature and the intent
was to give users good default behavior if they didn't include any
logging configuration on the classpath.

The problem with assuming that CL tooling is going to fix the job is
that many people link against spark as a library and run their
application using their own scripts. In this case the first thing
people see when they run an application that links against Spark was a
big ugly logging warning.

I'm not super familiar with log4j-over-slf4j, but this behavior of
returning null for the appenders seems a little weird. What is the use
case for using this and not just directly use slf4j-log4j12 like Spark
itself does?

Did you have a more general fix for this in mind? Or was your plan to
just revert the existing behavior... We might be able to add a
configuration option to disable this logging default stuff. Or we
could just rip it out - but I'd like to avoid that if possible.

- Patrick

On Thu, Feb 6, 2014 at 11:41 PM, Paul Brown <prb@mult.ifario.us> wrote:
> We have a few applications that embed Spark, and in 0.8.0 and 0.8.1, we
> were able to use slf4j, but 0.9.0 broke that and unintentionally forces
> direct use of log4j as the logging backend.
>
> The issue is here in the org.apache.spark.Logging trait:
>
> https://github.com/apache/incubator-spark/blame/master/core/src/main/scala/org/apache/spark/Logging.scala#L107
>
> log4j-over-slf4j *always* returns an empty enumeration for appenders to the
> ROOT logger:
>
> https://github.com/qos-ch/slf4j/blob/master/log4j-over-slf4j/src/main/java/org/apache/log4j/Category.java?source=c#L81
>
> And this causes an infinite loop and an eventual stack overflow.
>
> I'm happy to submit a Jira and a patch, but it would be significant enough
> reversal of recent changes that it's probably worth discussing before I
> sink a half hour into it.  My suggestion would be that initialization (or
> not) should be left to the user with reasonable default behavior supplied
> by the spark commandline tooling and not forced on applications that
> incorporate Spark.
>
> Thoughts/opinions?
>
> -- Paul
> --
> prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/

Mime
View raw message