From user-return-35655-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Jun 17 00:17:56 2015 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 611EF1837C for ; Wed, 17 Jun 2015 00:17:56 +0000 (UTC) Received: (qmail 80813 invoked by uid 500); 17 Jun 2015 00:17:52 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 80730 invoked by uid 500); 17 Jun 2015 00:17:52 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 80719 invoked by uid 99); 17 Jun 2015 00:17:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2015 00:17:52 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of saiph.kappa@gmail.com designates 74.125.82.45 as permitted sender) Received: from [74.125.82.45] (HELO mail-wg0-f45.google.com) (74.125.82.45) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2015 00:15:36 +0000 Received: by wgzl5 with SMTP id l5so24001891wgz.3 for ; Tue, 16 Jun 2015 17:17:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=0pWtxfxOGUwUnR6au7z70qNkem8Wr8TSGpeSW14ryFw=; b=BpJru+XGmcakMXjRjq88YzaKsCAjvc70N3SyFDL2BsYJFwRA4KSzPAOUzKvrMvsILH 3lLOtKZ48QgXUti1YNnoxXIO0dGBE/2VDtpyAcJhxJab8JXSl6GN1hBQaP6Xmj5uahF0 P1PER8QHmRywyHb0MLeHU3UJ46CIySOi9nricpldRcI9n9aKNSeumJH5imiRdDtiWHV7 aqg/6cXXnbAisQUS7i2++0GKtmO8UKv3J6qmXO+g0t2QI/RSd9Iy60TMeypYsnZ4zuOD xWQ+bPYEYtb7WYLRXaQygxpBmzy8yqh4qiPwjrDy1R7ZN7thr8Lpx/rPyFpfXIc49xyS yhvg== MIME-Version: 1.0 X-Received: by 10.180.84.170 with SMTP id a10mr12568957wiz.52.1434500244915; Tue, 16 Jun 2015 17:17:24 -0700 (PDT) Received: by 10.27.226.23 with HTTP; Tue, 16 Jun 2015 17:17:24 -0700 (PDT) Date: Wed, 17 Jun 2015 01:17:24 +0100 Message-ID: Subject: Unable to use more than 1 executor for spark streaming application with YARN From: Saiph Kappa To: user Content-Type: multipart/alternative; boundary=f46d041827e68749c80518ab9d74 X-Virus-Checked: Checked by ClamAV on apache.org --f46d041827e68749c80518ab9d74 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I am running a simple spark streaming application on hadoop 2.7.0/YARN (master: yarn-client) with 2 executors in different machines. However, while the app is running, I can see on the app web UI (tab executors) that only 1 executor keeps completing tasks over time, the other executor only works and completes tasks for some seconds. From the logs I can see an exception arising, though it is not clear what went wrong. Here is the yarn-nodemanager log: =C2=AB 2015-06-17 00:29:50,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Starting resource-monitoring for container_1434391147618_0007_01_000003 2015-06-17 00:29:50,977 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 286.5 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:29:53,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 463.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:29:57,009 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 465.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:00,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 467.6 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:03,032 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 474.0 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:06,041 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 480.2 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:09,053 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 540.9 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:12,068 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 550.9 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:15,075 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 551.1 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:18,090 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Memory usage of ProcessTree 30553 for container-id container_1434391147618_0007_01_000003: 558.7 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used 2015-06-17 00:30:20,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1434391147618_0007_01_000003 is : 1 2015-06-17 00:30:20,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1434391147618_0007_01_000003 and exit code: 1 ExitCodeException exitCode=3D1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCo= ntainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 615) at java.lang.Thread.run(Thread.java:745) 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1434391147618_0007_01_000003 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=3D1: 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456) 2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCo= ntainer(DefaultContainerExecutor.java:211) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch.call(ContainerLaunch.java:302) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch.call(ContainerLaunch.java:82) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 145) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 615) 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2015-06-17 00:30:20,158 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch: Container exited with a non-zero exit code 1 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Contai= nerImpl: Container container_1434391147618_0007_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE 2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch: Cleaning up container container_1434391147618_0007_01_000003 2015-06-17 00:30:20,178 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-myuser/nm-local-dir/usercache/myuser/appcache/application_14343= 91147618_0007/container_1434391147618_0007_01_000003 2015-06-17 00:30:20,178 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=3Dmyuser OPERATION=3DContainer Finished - Failed TARGET=3DContainerImpl RESULT=3DFAILURE DESCRIPTION=3DContainer failed with state: EXITED_WITH_FAILURE APPID=3Dapplication_1434391147618_0007 CONTAINERID=3Dcontainer_1434391147618_0007_01_000003 2015-06-17 00:30:20,178 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Contai= nerImpl: Container container_1434391147618_0007_01_000003 transitioned from EXITED_WITH_FAILURE to DONE 2015-06-17 00:30:20,179 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= icationImpl: Removing container_1434391147618_0007_01_000003 from application application_1434391147618_0007 2015-06-17 00:30:20,179 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1434391147618_0007 2015-06-17 00:30:20,500 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= icationImpl: Application application_1434391147618_0007 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2015-06-17 00:30:20,501 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-myuser/nm-local-dir/usercache/myuser/appcache/application_14343= 91147618_0007 2015-06-17 00:30:20,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1434391147618_0007 2015-06-17 00:30:20,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= icationImpl: Application application_1434391147618_0007 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2015-06-17 00:30:20,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAg= gregatingLogHandler: Scheduling Log Deletion for application: application_1434391147618_0007, with delay of 10800 seconds =C2=BB Not sure if it is relevant, but in the output of the application I keep getting this message: =C2=AB15/06/17 00:29:53 INFO ShuffledDStream: Time 1434497393000 ms is inva= lid as zeroTime is 1434497391000 ms and slideDuration is 4000 ms and difference is 2000 ms=C2=BB I'm using spark 1.3.2. Any ideas of what can be happening? Thanks. --f46d041827e68749c80518ab9d74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I am runnin= g a simple spark streaming application on hadoop 2.7.0/YARN (master: yarn-c= lient) with 2 executors in different machines. However, while the app is ru= nning, I can see on the app web UI (tab executors) that only 1 executor kee= ps completing tasks over time, the other executor only works and completes = tasks for some seconds. From the logs I can see an exception arising, thoug= h it is not clear what went wrong.

Here is the yar= n-nodemanager log:
=C2=AB
2015-06-17 00:29:50,967 INFO org.apache.had= oop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:= Starting resource-monitoring for container_1434391147618_0007_01_0000032015-06-17 00:29:50,977 INFO org.apache.hadoop.yarn.server.nodemanager.con= tainermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30= 553 for container-id container_1434391147618_0007_01_000003: 286.5 MB of 3 = GB physical memory used; 2.7 GB of 6.3 GB virtual memory used
2015-06-17= 00:29:53,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanag= er.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for con= tainer-id container_1434391147618_0007_01_000003: 463.7 MB of 3 GB physical= memory used; 2.7 GB of 6.3 GB virtual memory used
2015-06-17 00:29:57,0= 09 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.= ContainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id c= ontainer_1434391147618_0007_01_000003: 465.7 MB of 3 GB physical memory use= d; 2.7 GB of 6.3 GB virtual memory used
2015-06-17 00:30:00,024 INFO org= .apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersM= onitorImpl: Memory usage of ProcessTree 30553 for container-id container_14= 34391147618_0007_01_000003: 467.6 MB of 3 GB physical memory used; 2.7 GB o= f 6.3 GB virtual memory used
2015-06-17 00:30:03,032 INFO org.apache.had= oop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:= Memory usage of ProcessTree 30553 for container-id container_1434391147618= _0007_01_000003: 474.0 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB vi= rtual memory used
2015-06-17 00:30:06,041 INFO org.apache.hadoop.yarn.se= rver.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usa= ge of ProcessTree 30553 for container-id container_1434391147618_0007_01_00= 0003: 480.2 MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memor= y used
2015-06-17 00:30:09,053 INFO org.apache.hadoop.yarn.server.nodema= nager.containermanager.monitor.ContainersMonitorImpl: Memory usage of Proce= ssTree 30553 for container-id container_1434391147618_0007_01_000003: 540.9= MB of 3 GB physical memory used; 2.7 GB of 6.3 GB virtual memory used
2= 015-06-17 00:30:12,068 INFO org.apache.hadoop.yarn.server.nodemanager.conta= inermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 3055= 3 for container-id container_1434391147618_0007_01_000003: 550.9 MB of 3 GB= physical memory used; 2.7 GB of 6.3 GB virtual memory used
2015-06-17 0= 0:30:15,075 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager= .monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30553 for conta= iner-id container_1434391147618_0007_01_000003: 551.1 MB of 3 GB physical m= emory used; 2.7 GB of 6.3 GB virtual memory used
2015-06-17 00:30:18,090= INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Co= ntainersMonitorImpl: Memory usage of ProcessTree 30553 for container-id con= tainer_1434391147618_0007_01_000003: 558.7 MB of 3 GB physical memory used;= 2.7 GB of 6.3 GB virtual memory used
2015-06-17 00:30:20,157 WARN org.a= pache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code fr= om container container_1434391147618_0007_01_000003 is : 1
2015-06-17 00= :30:20,157 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerE= xecutor: Exception from container-launch with container ID: container_14343= 91147618_0007_01_000003 and exit code: 1
ExitCodeException exitCode=3D1:=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Sh= ell.runCommand(Shell.java:545)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.util.Shell.run(Shell.java:456)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Shell$ShellCommandExe= cutor.execute(Shell.java:722)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.laun= chContainer(DefaultContainerExecutor.java:211)
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager.containerma= nager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemana= ger.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.Futu= reTask.run(FutureTask.java:262)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1145)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.conc= urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:74= 5)
2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanage= r.ContainerExecutor: Exception from container-launch.
2015-06-17 00:30:2= 0,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Con= tainer id: container_1434391147618_0007_01_000003
2015-06-17 00:30:20,15= 7 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit co= de: 1
2015-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodeman= ager.ContainerExecutor: Stack trace: ExitCodeException exitCode=3D1:
201= 5-06-17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.Contain= erExecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.S= hell.runCommand(Shell.java:545)
2015-06-17 00:30:20,157 INFO org.apache.= hadoop.yarn.server.nodemanager.ContainerExecutor:=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at org.apache.hadoop.util.Shell.run(Shell.java:456)
2015-06= -17 00:30:20,157 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerEx= ecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Shell= $ShellCommandExecutor.execute(Shell.java:722)
2015-06-17 00:30:20,158 IN= FO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager.Defau= ltContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
2= 015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.Conta= inerExecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn= .server.nodemanager.containermanager.launcher.ContainerLaunch.call(Containe= rLaunch.java:302)
2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.se= rver.nodemanager.ContainerExecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at = org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contain= erLaunch.call(ContainerLaunch.java:82)
2015-06-17 00:30:20,158 INFO org.= apache.hadoop.yarn.server.nodemanager.ContainerExecutor:=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at java.util.concurrent.FutureTask.run(FutureTask.java:2= 62)
2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanag= er.ContainerExecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.conc= urrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-0= 6-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerE= xecutor:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.Thread= PoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-06-17 00:30:20= ,158 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745)<= br>2015-06-17 00:30:20,158 WARN org.apache.hadoop.yarn.server.nodemanager.c= ontainermanager.launcher.ContainerLaunch: Container exited with a non-zero = exit code 1
2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.n= odemanager.containermanager.container.ContainerImpl: Container container_14= 34391147618_0007_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE=
2015-06-17 00:30:20,158 INFO org.apache.hadoop.yarn.server.nodemanager.= containermanager.launcher.ContainerLaunch: Cleaning up container container_= 1434391147618_0007_01_000003
2015-06-17 00:30:20,178 INFO org.apache.had= oop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute pat= h : /tmp/hadoop-myuser/nm-local-dir/usercache/myuser/appcache/application_1= 434391147618_0007/container_1434391147618_0007_01_000003
2015-06-17 00:3= 0:20,178 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER= =3Dmyuser=C2=A0=C2=A0=C2=A0=C2=A0 OPERATION=3DContainer Finished - Failed= =C2=A0=C2=A0 TARGET=3DContainerImpl=C2=A0=C2=A0=C2=A0 RESULT=3DFAILURE=C2= =A0 DESCRIPTION=3DContainer failed with state: EXITED_WITH_FAILURE=C2=A0=C2= =A0=C2=A0 APPID=3Dapplication_1434391147618_0007=C2=A0=C2=A0=C2=A0 CONTAINE= RID=3Dcontainer_1434391147618_0007_01_000003
2015-06-17 00:30:20,178 INF= O org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Cont= ainerImpl: Container container_1434391147618_0007_01_000003 transitioned fr= om EXITED_WITH_FAILURE to DONE
2015-06-17 00:30:20,179 INFO org.apache.h= adoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:= Removing container_1434391147618_0007_01_000003 from application applicati= on_1434391147618_0007
2015-06-17 00:30:20,179 INFO org.apache.hadoop.yar= n.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP= for appId application_1434391147618_0007
2015-06-17 00:30:20,500 INFO o= rg.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appli= cationImpl: Application application_1434391147618_0007 transitioned from RU= NNING to APPLICATION_RESOURCES_CLEANINGUP
2015-06-17 00:30:20,501 INFO o= rg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting= absolute path : /tmp/hadoop-myuser/nm-local-dir/usercache/myuser/appcache/= application_1434391147618_0007
2015-06-17 00:30:20,501 INFO org.apache.h= adoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLI= CATION_STOP for appId application_1434391147618_0007
2015-06-17 00:30:20= ,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.applic= ation.ApplicationImpl: Application application_1434391147618_0007 transitio= ned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-06-17 00:30:2= 0,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logha= ndler.NonAggregatingLogHandler: Scheduling Log Deletion for application: ap= plication_1434391147618_0007, with delay of 10800 seconds
=C2=BB

=
Not sure if it is relevant, but in the output of the application I ke= ep getting this message:
=C2=AB15/06/17 00:29:53 INFO ShuffledDStream: T= ime 1434497393000 ms is invalid as zeroTime is 1434497391000 ms and slideDu= ration is 4000 ms and difference is 2000 ms=C2=BB

I'm usin= g spark 1.3.2.

Any ideas of what can be happening?

Thanks.

--f46d041827e68749c80518ab9d74--