spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiang, Yi J (CWM-NR)" <yi.j.ji...@rbc.com.INVALID>
Subject RE: Spark onApplicationEnd run multiple times during the application failure
Date Thu, 21 Nov 2019 13:54:43 GMT
Hello,
Thank you for replying, it is retry, but why retry is happened from whole application level?
To my understanding, the retry can be done in job level.
Jacky


From: hemant singh [mailto:hemant2184@gmail.com]
Sent: November 21, 19 3:12 AM
To: Jiang, Yi J (CWM-NR) <yi.j.jiang@rbc.com.invalid>
Cc: Martin, Phil <phil.martin@rbccm.com>; user@spark.apache.org
Subject: Re: Spark onApplicationEnd run multiple times during the application failure

Could it be because of re-try.

Thanks

On Thu, 21 Nov 2019 at 3:35 AM, Jiang, Yi J (CWM-NR) <yi.j.jiang@rbc.com.invalid<mailto:yi.j.jiang@rbc.com.invalid>>
wrote:
Hello
We are running into an issue.
We have customized the SparkListener class, and added that to spark context. But when the
spark job is failed, we find the “onApplicationEnd” function was triggered twice.
Is that supposed to be triggered just once when the spark job is failed? Because the application
level of spark job is only launched once, how can it be triggered twice when it is failed.
Please let us know
Thank you



_______________________________________________________________________

This email is intended only for the use of the individual(s) to whom it is addressed and may
be privileged and confidential.
Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please
advise immediately and delete the original message.
This message may have been altered without your or our knowledge and the sender does not accept
any liability for any errors or omissions in the message.

Ce courriel est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations
qui s'y rapportent.
Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par
une personne autre que le (les) destinataire(s) désigné(s) est interdite.
Si vous recevez ce courriel par erreur, veuillez m'en aviser immédiatement, par retour de
courriel ou par un autre moyen.
Mime
View raw message