spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hemant singh <hemant2...@gmail.com>
Subject Re: Spark onApplicationEnd run multiple times during the application failure
Date Thu, 21 Nov 2019 13:56:23 GMT
This is how it is. It is the whole application level retry after 4 tasks
attempts fail the whole application fails and then the application retry.

Thanks,
Hemant

On Thu, 21 Nov 2019 at 7:24 PM, Jiang, Yi J (CWM-NR) <yi.j.jiang@rbc.com>
wrote:

> Hello,
>
> Thank you for replying, it is retry, but why retry is happened from whole
> application level?
>
> To my understanding, the retry can be done in job level.
>
> Jacky
>
>
>
>
>
> *From:* hemant singh [mailto:hemant2184@gmail.com]
> *Sent:* November 21, 19 3:12 AM
> *To:* Jiang, Yi J (CWM-NR) <yi.j.jiang@rbc.com.invalid>
> *Cc:* Martin, Phil <phil.martin@rbccm.com>; user@spark.apache.org
> *Subject:* Re: Spark onApplicationEnd run multiple times during the
> application failure
>
>
>
> Could it be because of re-try.
>
>
>
> Thanks
>
>
>
> On Thu, 21 Nov 2019 at 3:35 AM, Jiang, Yi J (CWM-NR) <
> yi.j.jiang@rbc.com.invalid> wrote:
>
> Hello
>
> We are running into an issue.
>
> We have customized the SparkListener class, and added that to spark
> context. But when the spark job is failed, we find the “onApplicationEnd”
> function was triggered twice.
>
> Is that supposed to be triggered just once when the spark job is failed?
> Because the application level of spark job is only launched once, how can
> it be triggered twice when it is failed.
>
> Please let us know
>
> Thank you
>
>
>
>
>
> _______________________________________________________________________
>
> This email is intended only for the use of the individual(s) to whom it is
> addressed and may be privileged and confidential.
> Unauthorised use or disclosure is prohibited. If you receive this e-mail
> in error, please advise immediately and delete the original message.
> This message may have been altered without your or our knowledge and the
> sender does not accept any liability for any errors or omissions in the
> message.
>
> Ce courriel est confidentiel et protégé. L'expéditeur ne renonce pas aux
> droits et obligations qui s'y rapportent.
> Toute diffusion, utilisation ou copie de ce message ou des renseignements
> qu'il contient par une personne autre que le (les) destinataire(s)
> désigné(s) est interdite.
> Si vous recevez ce courriel par erreur, veuillez m'en aviser
> immédiatement, par retour de courriel ou par un autre moyen.
>
>

Mime
View raw message