spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "raymond.tan" <raymond.chiew....@gmail.com>
Subject Spark Event Log Forwarding and Offset Tracking
Date Wed, 13 Jan 2021 16:08:25 GMT
Hello here, I am new to spark and am trying to add some monitoring for spark
applications specifically to handle the below situations - 1 - Forwarding
Spark Event Logs to identify critical events like job start, executor
failures, job failures etc to ElasticSearch via log4j. However I could not
find any way to foward event log via log4j configurations. Is there any
other recommended approach to track these application events?2 - For Spark
streaming jobs, is there any way to identify that data from Kafka is not
consumed for whatever reason, or the offsets are not progressing as expected
and also forward that to ElasticSearch via log4j for
monitoringThanks,Raymond



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
Mime
View raw message