samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhijie Shen <>
Subject Re: Yarn High Availability - Samza Compatibility Plans
Date Thu, 11 Sep 2014 22:51:25 GMT
W.R.T HA RM, AM doesn't need to make any change if it is using
AMRMClient(Async), which will automatically take care of the work required
when RM failover happens.

One thing that may interest Samza is the work-preserving AM restarting. If
Samza AM crash for some reason, when it restarts, it can try to get the old
containers back, to continue the work. To achieve this, Samza needs some
logic optimization on the AM side.

On Thu, Sep 11, 2014 at 2:45 PM, Chris Riccomini <> wrote:

> Hey Ethan,
> Yes, we plan to support this, but haven't done any testing with it yet.
> The main question that I have is whether supporting HA RM requires Samza's
> AM to have code changes, or whether the YARN AM client will transparently
> handle RM failover. Until we run some tests on this (or are told
> otherwise), I just don't know (and haven't had time to investigate). Does
> anyone know the answer to this question?
> Cheers,
> Chris
> On 9/11/14 1:55 PM, "Ethan Setnik" <> wrote:
> >Yarn 2.4.0 brings support for high availability configurations by
> >specifying a cluster of resource managers and a state store via zookeeper.
> >
> >
> >"When there are multiple RMs, the configuration (yarn-site.xml) used by
> >clients and nodes is expected to list all the RMs. Clients,
> >ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs
> >in a round-robin fashion until they hit the Active RM. If the Active goes
> >down, they resume the round-robin polling until they hit the "new"
> >Active.²
> >
> >
> >Does Samza have any plans to support Yarn HA configurations?

Zhijie Shen
Hortonworks Inc.

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message