incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zheng <bearzheng2...@gmail.com>
Subject Deploy S4 App with Yarn -- Memory Problem
Date Thu, 15 Nov 2012 09:12:17 GMT
Hi All,

I am confused about the memory configuration.
In the S4ApplicationMaster.java, it says:
        // A resource ask has to be at least the minimum of the capability
of the cluster, the value has to be
        // a multiple of the min value and cannot exceed the max.
        // If it is not an exact multiple of min, the RM will allocate to
the nearest multiple of min
So I set the minimum memory as 128 in the yarn-site.xml
But when I deployed the Twitter Counter application, I got this error.

17:04:25.558 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
application report from ASM for, appId=1, clientToken=null,
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1352970249485, yarnAppState=RUNNING,
distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
17:04:26.560 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
application report from ASM for, appId=1, clientToken=null,
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1352970249485, yarnAppState=RUNNING,
distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
17:04:27.563 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
application report from ASM for, appId=1, clientToken=null,
appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0,
appStartTime=1352970249485, yarnAppState=RUNNING,
distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
application report from ASM for, appId=1, clientToken=null,
appDiagnostics=Application application_1352970173321_0001 failed 1 times
due to AM Container for appattempt_1352970173321_0001_000001 exited with
exitCode: 143 due to: Container
[pid=12119,containerID=container_1352970173321_0001_01_000001] is running
beyond physical memory limits. Current usage: 138.2mb of 128.0mb physical
memory used; 1.1gb of 5.2gb virtual memory used. Killing container.
Dump of the process-tree for container_1352970173321_0001_01_000001 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 12213 12119 12119 12119 (java) 285 11 1085763584 35085 java -Xmx128m
org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory 10
--num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
    |- 12119 9509 12119 12119 (bash) 4 3 108703744 304 /bin/bash -c java
-Xmx128m org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory 10
--num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
1>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stdout
2>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stderr



.Failing this attempt.. Failing the application., appMasterHost=,
appQueue=default, appMasterRpcPort=0, appStartTime=1352970249485,
yarnAppState=FAILED, distributedFinalState=FAILED, appTrackingUrl=,
appUser=root
17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Application
did not finish. YarnState=FAILED, DSFinalStatus=FAILED. Breaking monitoring
loop
17:04:28.569 [main] ERROR o.apache.s4.tools.yarn.S4YarnClient - Application
failed to complete successfully


Should the S4AppMaster use the multiple of minimum memory automatically?
Why the memory of container is only 128 MB?


Sincerely,
Zheng Yu
Mobile:  (852) 60670059
Email:    bearzheng2011@gmail.com

Mime
View raw message