spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Parhomenko <sparhome...@gmail.com>
Subject Unable to redirect Spark logs to slf4j
Date Wed, 05 Mar 2014 13:12:13 GMT
Hi,

I'm trying to redirect Spark logs to slf4j. Spark seem to be using Log4J,
so I did the typical steps of forcing a Log4J-based framework to use slf4j
- manually excluded slf4j-log4j12 and log4j, and included log4j-over-slf4j.
When doing that however Spark starts failing on initialization with:
java.lang.StackOverflowError
 at java.lang.ThreadLocal.access$400(ThreadLocal.java:72)
at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:376)
 at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:261)
at java.lang.ThreadLocal.get(ThreadLocal.java:146)
 at java.lang.StringCoding.deref(StringCoding.java:63)
at java.lang.StringCoding.encode(StringCoding.java:330)
 at java.lang.String.getBytes(String.java:916)
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
 at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.exists(File.java:813)
 at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1047)
 at sun.misc.URLClassPath.findResource(URLClassPath.java:176)
at java.net.URLClassLoader$2.run(URLClassLoader.java:551)
 at java.net.URLClassLoader$2.run(URLClassLoader.java:549)
at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findResource(URLClassLoader.java:548)
at java.lang.ClassLoader.getResource(ClassLoader.java:1147)
 at org.apache.spark.Logging$class.initializeLogging(Logging.scala:109)
at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:97)
 at org.apache.spark.Logging$class.log(Logging.scala:36)
at org.apache.spark.util.Utils$.log(Utils.scala:47)
 <last 4 lines repeated many, many times>

There's some related work done in
SPARK-1071<https://spark-project.atlassian.net/browse/SPARK-1071>,
but it was resolved after 0.9.0 was released. In the last comment Sean
refers to a StackOverflowError which was discussed in the mailing list, I
assume it might be a problem similar to mine but I was not able to find
that discussion.
Is anyone aware of a way to redirect Spark 0.9.0 logs to slf4j?

--
Best regards,
Sergey Parhomenko

Mime
View raw message