samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Tipper <john_tip...@hotmail.com>
Subject Re: Does Samza work with ResourceManager in HA?
Date Tue, 03 Nov 2015 16:04:59 GMT
Thanks Rick, much appreciated.  Did you have to set yarn.resourcemanager.hostname or can you
leave this out?

Thanks,

John

________________________________________
From: Rick Mangi <rick@chartbeat.com>
Sent: 03 November 2015 15:46
To: dev@samza.apache.org
Subject: Re: Does Samza work with ResourceManager in HA?

Hi John,

We just got this set up last week. I haven’t fully tested the failover but it certainly
works for testing out our samza jobs on a pretty large cluster.

All of the server names are fqdn in our live yarn-site.xml

Hope this helps,

Rick


<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>ops02</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>yarnmaster01</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>yarnmaster02</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>yarnmaster01:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>yarnmaster02:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-address</name>
  <value>zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk06:2181</va
lue>
</property>
<property>
  <name>yarn.resourcemanager.store.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>

<property>
  <name>yarn.resourcemanager.recovery.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.zk-state-store.parent-path</name>
  <value>/yarn_ops02_rmstore</value>
</property>



> On Nov 3, 2015, at 6:10 AM, John Tipper <john_tipper@hotmail.com> wrote:
>
>
>
> Does anyone have Samza working with resource manager in HA? If so, what do I set yarn.resourcemanager.hostname
to in yarn-site.xml?  Can anyone share a working yarn-site.xml please?
>
>
> If I set it to the first of my RMs, the the job submission works ok if I submit the job
from that RM and the RM is the active one. If the RM machine that I run the job submission
from is not active, I get connection refused errors on port 8032.  If I don't set it, I get
errors where run-job.sh tries to submit to 0.0.0.0:8032
>
>
> Many thanks,
>
>
> John


Mime
View raw message