spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shriraj Bhardwaj <>
Subject Spark ShutdownHook through python jobs.
Date Thu, 13 Aug 2020 11:31:54 GMT
we have spark jobs written totally in python similar to repo,
we are using spark-submit to submit the application in the local mode, but
want to send metrics when the job ends (on SIGTERM as well), to do so we
need something similar to shutdown hook in java/scala, can somebody help
how to do it ? I couldn't fine the shutdown hook interface in pyspark ?
Shriraj Bhardwaj

*कर्मण्येवाधिकारस्ते मा फलेषु कदाचन।
मा कर्मफलहेतुर्भूर्मा ते
सङ्गोऽस्त्वकर्मणि॥ *
View raw message