samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lukas Steiblys" <lu...@doubledutch.me>
Subject Re: Problems running new jobs in hello-samza
Date Mon, 06 Oct 2014 17:40:50 GMT
I'll add that if you check the YARN node application master container log 
and see that the job is constantly restarting, you might need to increase 
the container memory limit to 1024MB at least. Also, a good parameter to 
play with in YARN is  yarn.nodemanager.vmem-pmem-ratio.

Lukas

-----Original Message----- 
From: Mark Mindenhall
Sent: Monday, October 6, 2014 8:44 AM
To: dev@samza.incubator.apache.org
Subject: Re: Problems running new jobs in hello-samza

Hi Zach,

I’m also a relative newbie, but I did run into this same issue.  You are 
correct, in that your 5th job isn’t starting due to not enough resources 
available in the cluster, so you need to reduce the resources required.

First, in yarn-site.xml I switched over to the 
FairScheduler<http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>:

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>

I also added these two properties (yarn-site.xml) to control the amount of 
memory allocated to each job:

  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
    <description>Minimum limit of memory to allocate to each container 
request at the Resource Manager.</description>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>512</value>
    <description>Maximum limit of memory to allocate to each container 
request at the Resource Manager.</description>
  </property>

Then, in each of my Samza properties files describing my jobs, I added the 
following two settings:

    yarn.container.memory.mb=512
    yarn.am.container.memory.mb=256

Hope that helps!

Best,
Mark


On Oct 6, 2014, at 6:27 AM, Zach Cox 
<zcox522@gmail.com<mailto:zcox522@gmail.com>> wrote:

Hi - I'm just getting started with Samza. I got the hello-samza example
working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
everything and submitted them to yarn using run-job.sh. These 2 new jobs
show up in the yarn web ui, however only one of them has State=RUNNING, the
other just sits forever at State=ACCEPTED.

The Cluster Metrics section shows some interesting things:
- Apps Pending = 1
- Apps Running = 4
- Containers Running = 8
- Memory Used = 8 GB
- Memory Total = 8 GB
- Memory Reserved = 0 B

Again I'm really new to samza & yarn, but does this mean that the node on
this vagrant box has 8 GB memory available but all 8 GB is being used, so
it can't run the 5th samza job?

Are there 8 containers running because each Samza job has an
ApplicationMaster and a SamzaContainer? Are each of those containers using
1 GB memory, and that's why all the available memory is used up? Do these
containers really need 1 GB memory each? Can this be adjusted somehow?

Just trying to better understand what's going on here, and see if there's a
simple way to get both of my new tasks running in hello-samza.

Thanks,
Zach


Mime
View raw message