spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivani Rao <>
Subject using Log4j to log INFO level messages on workers
Date Mon, 02 Jun 2014 23:18:20 GMT
Hello Spark fans,

I am trying to log messages from my spark application. When the main()
function attempts to log, using it works great, but when I try
the same command from the code that probably runs on the worker, I
initially got an serialization error. To solve that, I created a new logger
in the code that operates on the data, which solved the serialization issue
but now there is no output in the console or on the worker node logs. I
don't see any application level log messages in the spark logs either. When
I use println() instead, I do see console output being  generated.

I tried the following and none of them works

a) pass by using in my java command
line initiation of the spark application
b) setting the properties within the worker by calling log.addAppender(new

None of them work.

What am i missing?

Software Engineer
Analytics Engineering Team@ Box
Mountain View, CA

View raw message