spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Spark Event Log Forwarding and Offset Tracking
Date Sun, 17 Jan 2021 14:08:55 GMT
Hi,

> Forwarding Spark Event Logs to identify critical events like job start,
executor failures, job failures etc to ElasticSearch via log4j. However I
could not find any way to foward event log via log4j configurations. Is
there any other recommended approach to track these application events?

I'd use SparkListener API (
http://spark.apache.org/docs/latest/api/scala/org/apache/spark/scheduler/SparkListener.html
)

> 2 - For Spark streaming jobs, is there any way to identify that data from
Kafka is not consumed for whatever reason, or the offsets are not
progressing as expected and also forward that to ElasticSearch via log4j
for monitoring

Think SparkListener API would help here too.

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski

<https://twitter.com/jaceklaskowski>


On Wed, Jan 13, 2021 at 5:15 PM raymond.tan <raymond.chiew.tan@gmail.com>
wrote:

> Hello here, I am new to spark and am trying to add some monitoring for
> spark applications specifically to handle the below situations - 1 -
> Forwarding Spark Event Logs to identify critical events like job start,
> executor failures, job failures etc to ElasticSearch via log4j. However I
> could not find any way to foward event log via log4j configurations. Is
> there any other recommended approach to track these application events? 2 -
> For Spark streaming jobs, is there any way to identify that data from Kafka
> is not consumed for whatever reason, or the offsets are not progressing as
> expected and also forward that to ElasticSearch via log4j for monitoring
> Thanks, Raymond
> ------------------------------
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>

Mime
View raw message