spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naved Alam <alam.na...@outlook.com>
Subject [Spark-Listener] [How-to] Listen only to specific events
Date Tue, 13 Feb 2018 08:15:51 GMT
I have a spark application which creates multiple sessions. Each of these sessions can run
jobs in parallel. I want to log some details about the execution of these jobs, but want to
the tag them with the session they were called from.

I tried creating a listener from within each session (instantiations of the same class, but
with session specific information), but whenever an event is emmited, it is intercepted by
all the listeners (which makes sense since the listeners are attached to the shared context).
Is there a way I can ensure that the events are only intercepted by the listener from the
same session. If not, then is it possible to add any identifiers to the events which can be
used to figure the session which triggered a job?

Thanks
Naved

Mime
View raw message