spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Gaudio <>
Subject Re: using Log4j to log INFO level messages on workers
Date Tue, 03 Jun 2014 05:33:06 GMT

I had the same problem with pyspark.  Here's how I resolved it:

What I've found in python (not sure about scala) is that if the function
being serialized was written in the same python module as the main
function, then logging fails.  If the serialized function is in a separate
module, then logging does not fail.  I just created this gist to demo the
situation and (python) solution.  Is there a similar way to do this in


On Mon, Jun 2, 2014 at 7:18 PM, Shivani Rao <> wrote:

> Hello Spark fans,
> I am trying to log messages from my spark application. When the main()
> function attempts to log, using it works great, but when I try
> the same command from the code that probably runs on the worker, I
> initially got an serialization error. To solve that, I created a new logger
> in the code that operates on the data, which solved the serialization issue
> but now there is no output in the console or on the worker node logs. I
> don't see any application level log messages in the spark logs either. When
> I use println() instead, I do see console output being  generated.
> I tried the following and none of them works
> a) pass by using in my java command
> line initiation of the spark application
> b) setting the properties within the worker by calling log.addAppender(new
> ConsoleAppender)
> None of them work.
> What am i missing?
> Thanks,
> Shivani
> --
> Software Engineer
> Analytics Engineering Team@ Box
> Mountain View, CA

View raw message