spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From QiaoanChen <kazenoyumec...@gmail.com>
Subject Logging problem in Spark when using Flume Log4jAppender
Date Tue, 18 Nov 2014 08:12:40 GMT
Hi,
I want to do log aggregation in a Spark standalone mode cluster, using
Apache Flume. But something weird happended. Here are my operations:

(1) Start a flume agent, listening on port 33333. ( flume-conf.properties
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/flume-conf.properties>

)
(2) In log4j.properties, I added a Flume Log4jAppender to make Spark able to
send its log to a flume agent. 
     ( log4j.properties
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/log4j.properties> 
)
(3) sbin/start-all.sh. 
     In this stage, all logs were aggregated into the flume agent and no
exception happened. 
     (flume-ng-log4jappender version: 1.5.0.1)
(4) bin/spark-submit --class org.apache.examples.SparkPi
../lib/spark-examples-1.1.0-hadoop2.4.0.jar 1
     At the beginning, logs were aggregated normally. However, when the
driver need to stop SparkContext sc, it threw some exceptions and the driver
exit abnormally:

sparkPi.log
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi.log>  
/====
...
14/11/18 23:06:24 INFO
SparkDeploySchedulerBackend:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- Asking each executor to shut down
14/11/18 23:06:25 INFO
MapOutputTrackerMasterActor:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- MapOutputTrackerActor stopped!
14/11/18 23:06:25 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Selector thread was interrupted!
log4j:ERROR Flume append() failed.
14/11/18 23:06:25 ERROR
ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:96)
- Error in select loop
org.apache.flume.FlumeException: Flume append() failed. Exception follows.
...
====/


I also tried setting log4j.appender.flume.unsafeMode property to true. This
time, the drive finished the job but didn't exit:

sparkPi_snd.log
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi_snd.log>  
/====
....
14/11/18 23:36:49 WARN
NettyAvroRpcClient:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- Using default maxIOWorkers
log4j:ERROR Cannot Append to Appender! Appender either closed or not setup
correctly!
14/11/18 23:36:49 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Removing ReceivingConnection to ConnectionManagerId(pc215,59272)
14/11/18 23:36:49 ERROR
ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:75)
- Corresponding SendingConnection to ConnectionManagerId(pc215,59272) not
found
14/11/18 23:36:49 WARN
ConnectionManager:org.apache.spark.Logging$class.logWarning(Logging.scala:71)
- All connections not cleaned up
14/11/18 23:36:49 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
ConnectionManager stopped
14/11/18 23:36:49 INFO
MemoryStore:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
MemoryStore cleared
14/11/18 23:36:49 INFO
BlockManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
BlockManager stopped
14/11/18 23:36:49 INFO
BlockManagerMaster:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- BlockManagerMaster stopped
14/11/18 23:36:49 INFO
RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Shutting down remote daemon.
14/11/18 23:36:49 INFO
RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Remote daemon shut down; proceeding with flushing remote transports.
14/11/18 23:36:49 INFO
Remoting:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Remoting shut down
14/11/18 23:36:49 INFO
SparkContext:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Successfully stopped SparkContext
====/


I have searched Google with keywords like "Spark log aggregation", "Spark
Flume Log4jAppender", "ERROR Flume append() failed", but I can't figure out
how to solve this problem.
Does anybody has an idea?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Logging-problem-in-Spark-when-using-Flume-Log4jAppender-tp19140.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message