incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gómez Ferro <danie...@yahoo-inc.com>
Subject Re: Deploy S4 App with Yarn -- Memory Problem
Date Fri, 16 Nov 2012 14:52:25 GMT
Hi,

It would be great if you could attach the logs for the containers 
themselves (not for the Application Master).

You can go to the YARN web interface, go to the Nodes page, for each 
node go to their containers page, and attach the log for the container 
that exited early.

Thanks!

Daniel


On 16/11/12 03:23 , Frank Zheng wrote:
> Hi,
>
> I got another problem when running the Twitter Trend Example in cluster
> mode.
> I can deploy the twitter-counter and twitter-adapter.
> For adapter, there are 2 containers running, one is the AppMaster and
> the other is the adapter application.
> For counter, there are also 2 containers running, one is the AppMaster
> and the other is the counter application.
> There should be one more container running for counter application, yes?
> I check the logs of the AppMaster of twitter-counter.
> Two containers for counter application were set up, but one of them was
> completed. I don't know why.
> Is this problem related to memory allocation?
>
> 10:10:55.925 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Initializing ApplicationMaster
with args [--container_memory, 10, --num_containers, 2, --priority, 0, -c, counter, -zk, testing.machine1:2181]
>
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Application master for
app, appId=1, clustertimestamp=1353031752144, attemptId=1
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Application master args
= []
>
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg / Amount of memory
in MB to be requested to host the S4 node / container_memory / 10
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg / No. of containers
on which the S4 node needs to be hosted / num_containers / 2
>
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg / Application Priority.
Default 0 / priority / 0
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg / The cluster (or
application) name / c / counter
>
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg / Zookeeper connection
string / zk / testing.machine1:2181
> 10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Application master args
= [[ option: container_memory  [ARG] :: Amount of memory in MB to be requested to host the
S4 node ], [ option: num_containers  [ARG] :: No. of containers on which the S4 node needs
to be hosted ], [ option: priority  [ARG] :: Application Priority. Default 0 ], [ option:
c cluster  [ARG] :: The cluster (or application) name ], [ option: zk  [ARG] :: Zookeeper
connection string ]]
>
> 10:10:56.111 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Starting ApplicationMaster
> 10:10:56.209 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Connecting to ResourceManager
at testing.machine1/192.168.0.100:8030  <http://192.168.0.100:8030>
>
> 10:10:56.672 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Min mem capability of
resources in this cluster 512
> 10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Max mem capability of
resources in this cluster 3072
>
> 10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Container memory specified
below min threshold of cluster. Using min value., specified=10, min=512
> 10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=0, appDone=false, total=2, requested=0, completed=0, failed=0, currentAllocated=0
>
> 10:10:57.682 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=2
> 10:10:57.684 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=1, releasedSet=0, progress=0.0
>
> 10:10:57.684 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Requested container
ask: {Priority: {Priority: 0}, Capability: memory: 512}
> 10:10:57.698 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:10:57.698 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 5632
> 10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=0, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=0
> 10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=1, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=0
>
> 10:10:58.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:10:58.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:10:58.715 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=2
> 10:10:58.717 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Launching S4 node on
a new container., containerId=container_1353031752144_0001_01_000002, containerNode=testing.machine1:40558,
containerNodeURI=testing.machine1:8042, containerStateNEW, containerResourceMemory512
>
> 10:10:58.719 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Launching S4 node on
a new container., containerId=container_1353031752144_0001_01_000003, containerNode=testing.machine2:44677,
containerNodeURI=testing.machine2:8042, containerStateNEW, containerResourceMemory512
>
> 10:10:58.719 [Thread-2] DEBUG o.a.s.tools.yarn.S4ApplicationMaster - Connecting to ContainerManager
for containerid=container_1353031752144_0001_01_000002
> 10:10:58.719 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Connecting to ContainerManager
at testing.machine1:40558
>
> 10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=1, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=2, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:10:58.726 [Thread-3] DEBUG o.a.s.tools.yarn.S4ApplicationMaster - Connecting to ContainerManager
for containerid=container_1353031752144_0001_01_000003
> 10:10:58.726 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Connecting to ContainerManager
at testing.machine2:44677
>
> 10:10:58.735 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Setting up container
launch container for containerid=container_1353031752144_0001_01_000003
> 10:10:58.735 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Setting up container
launch container for containerid=container_1353031752144_0001_01_000002
>
> 10:10:58.740 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Using default user
name root
> 10:10:58.740 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Using default user
name root
> 10:10:59.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:10:59.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=2, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=3, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:00.729 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:00.729 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=3, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=4, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:01.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:01.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=4, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=5, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:02.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:02.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=5, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=6, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:03.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:03.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:03.738 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=6, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=7, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:04.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:04.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=7, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=8, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:05.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:05.743 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=8, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=9, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:06.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:06.746 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=9, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=10, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:07.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:07.749 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:07.753 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=10, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=11, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:08.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:08.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:08.756 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=11, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=12, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:09.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:09.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:09.764 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=12, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=13, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:10.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:10.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=13, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=14, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:11.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:11.769 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=14, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:11.773 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=15, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:12.773 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:12.774 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:12.776 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=15, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=16, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:13.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:13.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
> 10:11:13.779 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=16, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
> 10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=17, appDone=false, total=2, requested=2, completed=0, failed=0, currentAllocated=2
>
> 10:11:14.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:14.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.0
>
> 10:11:14.782 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:14.782 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=1
> 10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got container status
for containerID= container_1353031752144_0001_01_000003, state=COMPLETE, exitStatus=0, diagnostics=
>
> 10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Container completed
successfully., containerId=container_1353031752144_0001_01_000003
> 10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=17, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
> 10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=18, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
> 10:11:15.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:15.784 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.5
> 10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=18, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
> 10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=19, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
> 10:11:16.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:16.787 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.5
>
> 10:11:16.789 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=19, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
> 10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=20, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
> 10:11:17.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
>
> 10:11:17.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.5
> 10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
>
> 10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
> 10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
>
> 10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=20, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
> 10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=21, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
> 10:11:18.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Asking RM for containers,
askCount=0
> 10:11:18.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Sending request to RM
for containers, requestedSet=0, releasedSet=0, progress=0.5
>
> 10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, allocatedCnt=0
> 10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current available resources
in the cluster memory: 4608
>
> 10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got response from RM
for container ask, completedCnt=0
> 10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=21, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
> 10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Current application
state: loop=22, appDone=false, total=2, requested=2, completed=1, failed=0, currentAllocated=2
>
>
>
> On Fri, Nov 16, 2012 at 2:32 AM, Matthieu Morel
> <matthieu.morel@gmail.com <mailto:matthieu.morel@gmail.com>> wrote:
>
>     Hi,
>
>     What specific parameter are you trying to set in the yarn
>     configuration? It looks like you have a max of 128MB per container.
>
>     What seems to be failing is the application master. It uses (by
>     default) 128MB for heap + default stack size, but that would be over
>     128MB.
>
>     Of course, we still have to properly handle all  parameters available in
>     the yarn command, including memory settings, but from what I see the
>     issue you are facing is related to the configuration of your Yarn
>     cluster. You can get more information from other log files (apart from
>     the client console logs), maybe that can help as well. But somewhere
>     you seem to be capping the container memory to an insufficient value.
>
>     Regards,
>
>     Matthieu
>
>     On Thu, Nov 15, 2012 at 9:12 AM, Frank Zheng
>     <bearzheng2011@gmail.com <mailto:bearzheng2011@gmail.com>> wrote:
>      > Hi All,
>      >
>      > I am confused about the memory configuration.
>      > In the S4ApplicationMaster.java, it says:
>      >         // A resource ask has to be at least the minimum of the
>     capability
>      > of the cluster, the value has to be
>      >         // a multiple of the min value and cannot exceed the max.
>      >         // If it is not an exact multiple of min, the RM will
>     allocate to
>      > the nearest multiple of min
>      > So I set the minimum memory as 128 in the yarn-site.xml
>      > But when I deployed the Twitter Counter application, I got this
>     error.
>      >
>      > 17:04:25.558 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
>      > application report from ASM for, appId=1, clientToken=null,
>     appDiagnostics=,
>      > appMasterHost=, appQueue=default, appMasterRpcPort=0,
>      > appStartTime=1352970249485, yarnAppState=RUNNING,
>      > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
>      > 17:04:26.560 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
>      > application report from ASM for, appId=1, clientToken=null,
>     appDiagnostics=,
>      > appMasterHost=, appQueue=default, appMasterRpcPort=0,
>      > appStartTime=1352970249485, yarnAppState=RUNNING,
>      > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
>      > 17:04:27.563 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
>      > application report from ASM for, appId=1, clientToken=null,
>     appDiagnostics=,
>      > appMasterHost=, appQueue=default, appMasterRpcPort=0,
>      > appStartTime=1352970249485, yarnAppState=RUNNING,
>      > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
>      > 17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
>      > application report from ASM for, appId=1, clientToken=null,
>      > appDiagnostics=Application application_1352970173321_0001 failed
>     1 times due
>      > to AM Container for appattempt_1352970173321_0001_000001 exited with
>      > exitCode: 143 due to: Container
>      > [pid=12119,containerID=container_1352970173321_0001_01_000001] is
>     running
>      > beyond physical memory limits. Current usage: 138.2mb of 128.0mb
>     physical
>      > memory used; 1.1gb of 5.2gb virtual memory used. Killing container.
>      > Dump of the process-tree for container_1352970173321_0001_01_000001 :
>      >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>      > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
>     FULL_CMD_LINE
>      >     |- 12213 12119 12119 12119 (java) 285 11 1085763584 35085
>     java -Xmx128m
>      > org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory 10
>      > --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
>      >     |- 12119 9509 12119 12119 (bash) 4 3 108703744 304 /bin/bash
>     -c java
>      > -Xmx128m org.apache.s4.tools.yarn.S4ApplicationMaster
>     --container_memory 10
>      > --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
>      >
>     1>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stdout
>      >
>     2>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stderr
>      >
>      >
>      > .Failing this attempt.. Failing the application., appMasterHost=,
>      > appQueue=default, appMasterRpcPort=0, appStartTime=1352970249485,
>      > yarnAppState=FAILED, distributedFinalState=FAILED, appTrackingUrl=,
>      > appUser=root
>      > 17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient -
>     Application
>      > did not finish. YarnState=FAILED, DSFinalStatus=FAILED. Breaking
>     monitoring
>      > loop
>      > 17:04:28.569 [main] ERROR o.apache.s4.tools.yarn.S4YarnClient -
>     Application
>      > failed to complete successfully
>      >
>      >
>      > Should the S4AppMaster use the multiple of minimum memory
>     automatically? Why
>      > the memory of container is only 128 MB?
>      >
>      >
>      > Sincerely,
>      > Zheng Yu
>      > Mobile:  (852) 60670059
>      > Email: bearzheng2011@gmail.com <mailto:bearzheng2011@gmail.com>
>      >
>      >
>      >
>
>
>
>
> --
> Sincerely,
> Zheng Yu
> Mobile:  (852) 60670059
> Email: bearzheng2011@gmail.com <mailto:bearzheng2011@gmail.com>
>
>
>

Mime
View raw message