samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <>
Subject Re: Yarn High Availability - Samza Compatibility Plans
Date Fri, 12 Sep 2014 14:57:46 GMT
Hey Zhijie,

Thanks, that's what I suspected. We are using the AMRMClient(Async), so if
you're using YARN 2.4.0 with HA RM, Samza should just work.

Regarding work-preserving AM, here's the ticket that's tracking this
feature in Samza:


On 9/11/14 3:51 PM, "Zhijie Shen" <> wrote:

>W.R.T HA RM, AM doesn't need to make any change if it is using
>AMRMClient(Async), which will automatically take care of the work required
>when RM failover happens.
>One thing that may interest Samza is the work-preserving AM restarting. If
>Samza AM crash for some reason, when it restarts, it can try to get the
>containers back, to continue the work. To achieve this, Samza needs some
>logic optimization on the AM side.
>On Thu, Sep 11, 2014 at 2:45 PM, Chris Riccomini <
>> wrote:
>> Hey Ethan,
>> Yes, we plan to support this, but haven't done any testing with it yet.
>> The main question that I have is whether supporting HA RM requires
>> AM to have code changes, or whether the YARN AM client will
>> handle RM failover. Until we run some tests on this (or are told
>> otherwise), I just don't know (and haven't had time to investigate).
>> anyone know the answer to this question?
>> Cheers,
>> Chris
>> On 9/11/14 1:55 PM, "Ethan Setnik" <> wrote:
>> >Yarn 2.4.0 brings support for high availability configurations by
>> >specifying a cluster of resource managers and a state store via
>> >
>> >
>> >"When there are multiple RMs, the configuration (yarn-site.xml) used by
>> >clients and nodes is expected to list all the RMs. Clients,
>> >ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the
>> >in a round-robin fashion until they hit the Active RM. If the Active
>> >down, they resume the round-robin polling until they hit the "new"
>> >Active.²
>> >
>> >
>> >Does Samza have any plans to support Yarn HA configurations?
>Zhijie Shen
>Hortonworks Inc.
>NOTICE: This message is intended for the use of the individual or entity
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>and delete it from your system. Thank You.

View raw message