hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "epayne@apache.org" <epa...@apache.org>
Subject Re: Application Master High Availability
Date Wed, 12 Aug 2020 13:50:25 GMT
Bharath,
I just want to clarify a couple of things. The yarn distributed shell is its own framework,
and it does support preserving containers across AM restart. I have tested this. But the MapReduce
framework does not support this feature (see https://issues.apache.org/jira/browse/MAPREDUCE-6608).
I spoke with Jon Eagles, and he believes that the Tez framework does support the container-preserving
feature.

On Tuesday, August 11, 2020, 5:53:32 PM CDT, Bharath Kumara Subramanian <codin.martial@gmail.com>
wrote: 

Thanks Eric & Wilfred.
To give you some context, I work on Apache Samza and we have streaming as a
service offering on top of YARN.
Ideally, we would like to ensure in the event of AM restarts, the container
it spawned can still be taken over by the new AM.

"Keep containers across application attempts" seems like an option that
might work for us.
Let me investigate and play with the parameter.

Appreciate your quick response.

Cheers,
Bharath


On Tue, Aug 11, 2020 at 6:31 AM Eric Payne <eric.payne1000@yahoo.com> wrote:

> Bharath,
>
> while there is no concept of HA AM in YARN, some frameworks do support
> preserving containers across AM restarts.
> In the yarn distributed shell, for example, you can set the
> "-keep_containers_across_application_attempts"
> parameter.
>
> -Eric
>
>
> On Monday, August 10, 2020, 7:18:18 PM CDT, Bharath Kumara Subramanian <
> codin.martial@gmail.com> wrote:
>
> Hi,
>
> I am looking for more documentation/information on AM high availability. I
> looked through the documentation and found resources on RM high
> availability but none for AM.
>
> I understand, YARN has provisions to restart AM in case of failure up to a
> configured number of attempts. However, I wanted to know if YARN has an
> active/standby option for AM. I would like to avoid bringing up the
> processing containers again in the event of AM failure and have the standby
> AM take over the managing my application.
>
> Thanks,
> Bharath
>

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message