spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chandan prakash <chandanbaran...@gmail.com>
Subject spark streaming: issue with logging with separate log4j properties files for driver and executor
Date Mon, 23 May 2016 11:48:18 GMT
Hi,
I am able to do logging for driver but not for executor.

I am running spark streaming under mesos.
Want to do log4j logging separately for driver and executor.

Used the below option in spark-submit command :
--driver-java-options
"-Dlog4j.configuration=file:/usr/local/spark-1.5.1-bin-hadoop2.6/conf/log4j_RequestLogDriver.properties"
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:
/usr/local/spark-1.5.1-bin-hadoop2.6/conf/log4j_RequestLogExecutor.properties
"

Logging for driver at path mentioned as in
log4j_RequestLogDriver.properties(/tmp/requestLogDriver.log) is happening
fine.
But for executor, there is no logging happening (shud be at
/tmp/requestLogExecutor.log as mentioned in log4j_RequestLogExecutor.properties
on executor machines)

*Any suggestions how to get logging enabled for executor ?*

TIA,
Chandan

-- 
Chandan Prakash

Mime
View raw message