spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivani Rao <>
Subject Re: using Log4j to log INFO level messages on workers
Date Wed, 04 Jun 2014 16:46:16 GMT
Hello Alex

Thanks for the link. Yes creating a singleton object for logging outside
the code that gets executed on the workers definitely works. The problem
that i am facing though is related to configuration of the logger. I don't
see any log messages in the worker logs of the application.

a) when i use println, I see the messages from the worker being logged into
the main driver of the application
b) when i use logger,  i see logger messages from the main() but not from
the workers.

Maybe I should upload a MWE (minimum working example) to demonstrate my


On Mon, Jun 2, 2014 at 10:33 PM, Alex Gaudio <> wrote:

> Hi,
> I had the same problem with pyspark.  Here's how I resolved it:
> What I've found in python (not sure about scala) is that if the function
> being serialized was written in the same python module as the main
> function, then logging fails.  If the serialized function is in a separate
> module, then logging does not fail.  I just created this gist to demo the
> situation and (python) solution.  Is there a similar way to do this in
> scala?
> Alex
> On Mon, Jun 2, 2014 at 7:18 PM, Shivani Rao <> wrote:
>> Hello Spark fans,
>> I am trying to log messages from my spark application. When the main()
>> function attempts to log, using it works great, but when I
>> try the same command from the code that probably runs on the worker, I
>> initially got an serialization error. To solve that, I created a new logger
>> in the code that operates on the data, which solved the serialization issue
>> but now there is no output in the console or on the worker node logs. I
>> don't see any application level log messages in the spark logs either. When
>> I use println() instead, I do see console output being  generated.
>> I tried the following and none of them works
>> a) pass by using in my java command
>> line initiation of the spark application
>> b) setting the properties within the worker by
>> calling log.addAppender(new ConsoleAppender)
>> None of them work.
>> What am i missing?
>> Thanks,
>> Shivani
>> --
>> Software Engineer
>> Analytics Engineering Team@ Box
>> Mountain View, CA

Software Engineer
Analytics Engineering Team@ Box
Mountain View, CA

View raw message