samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Shaw <jor...@pubnub.com>
Subject Re: Does Samza work with ResourceManager in HA?
Date Tue, 03 Nov 2015 18:55:22 GMT
John, we ran samza in HA. Something key to point out is the failover is not
graceful once zookeeper (assuming that's your provider) elects the standby
resourcemanager that resource manager will tear down and restart all
applications, and so depending on how long it takes your jobs to bootstrap
you will probably have minutes of downtime ... just heads up.
-Jordan

On Tue, Nov 3, 2015 at 8:04 AM, John Tipper <john_tipper@hotmail.com> wrote:

> Thanks Rick, much appreciated.  Did you have to set
> yarn.resourcemanager.hostname or can you leave this out?
>
> Thanks,
>
> John
>
> ________________________________________
> From: Rick Mangi <rick@chartbeat.com>
> Sent: 03 November 2015 15:46
> To: dev@samza.apache.org
> Subject: Re: Does Samza work with ResourceManager in HA?
>
> Hi John,
>
> We just got this set up last week. I haven’t fully tested the failover but
> it certainly works for testing out our samza jobs on a pretty large cluster.
>
> All of the server names are fqdn in our live yarn-site.xml
>
> Hope this helps,
>
> Rick
>
>
> <property>
>   <name>yarn.resourcemanager.ha.enabled</name>
>   <value>true</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
>   <value>true</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.cluster-id</name>
>   <value>ops02</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.ha.rm-ids</name>
>   <value>rm1,rm2</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.hostname.rm1</name>
>   <value>yarnmaster01</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.hostname.rm2</name>
>   <value>yarnmaster02</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.webapp.address.rm1</name>
>   <value>yarnmaster01:8088</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.webapp.address.rm2</name>
>   <value>yarnmaster02:8088</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.zk-address</name>
>   <value>zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk06:2181</va
> lue>
> </property>
> <property>
>   <name>yarn.resourcemanager.store.class</name>
>
> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
> </property>
>
> <property>
>   <name>yarn.resourcemanager.recovery.enabled</name>
>   <value>true</value>
> </property>
> <property>
>   <name>yarn.resourcemanager.zk-state-store.parent-path</name>
>   <value>/yarn_ops02_rmstore</value>
> </property>
>
>
>
> > On Nov 3, 2015, at 6:10 AM, John Tipper <john_tipper@hotmail.com> wrote:
> >
> >
> >
> > Does anyone have Samza working with resource manager in HA? If so, what
> do I set yarn.resourcemanager.hostname to in yarn-site.xml?  Can anyone
> share a working yarn-site.xml please?
> >
> >
> > If I set it to the first of my RMs, the the job submission works ok if I
> submit the job from that RM and the RM is the active one. If the RM machine
> that I run the job submission from is not active, I get connection refused
> errors on port 8032.  If I don't set it, I get errors where run-job.sh
> tries to submit to 0.0.0.0:8032
> >
> >
> > Many thanks,
> >
> >
> > John
>
>


-- 
Jordan Shaw
Full Stack Software Engineer
PubNub Inc
1045 17th St
San Francisco, CA 94107

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message