spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Friedman <eric.d.fried...@gmail.com>
Subject Re: pyspark yarn got exception
Date Fri, 05 Sep 2014 13:41:52 GMT
Are you certain the executors are using the same python?  What is in PYSPARK_PYTHON?  

I had a similar issue that arose from using one python on the driver's path and a different one on the cluster 

It can be useful to create a string that has a small python program to describe the environment on the workers and
Then eval that string in a distributed map function and collect the result. 

----
Eric Friedman

> On Sep 3, 2014, at 10:15 PM, Oleg Ruchovets <oruchovets@gmail.com> wrote:
> 
> Hi Andrew.
> 
> Problem still occur: 
> 
> all machines are using python 2.7:
> 
> [root@HDOP-N2 conf]# python --version
> Python 2.7.7 :: Anaconda 2.0.1 (64-bit)
> 
> Executing command from bin/pyspark:
>            [root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# bin/pyspark    --driver-memory 4g --executor-memory 2g --executor-cores 1       examples/src/main/python/pi.py   1000
> 
> 
> Python 2.7.7 |Anaconda 2.0.1 (64-bit)| (default, Jun  2 2014, 12:34:02) 
> [GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> Anaconda is brought to you by Continuum Analytics.
> Please check out: http://continuum.io/thanks and https://binstar.org
> Traceback (most recent call last):
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/shell.py", line 43, in <module>
>     sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 94, in __init__
>     SparkContext._ensure_initialized(self, gateway=gateway)
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 190, in _ensure_initialized
>     SparkContext._gateway = gateway or launch_gateway()
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/java_gateway.py", line 51, in launch_gateway
>     gateway_port = int(proc.stdout.readline())
> ValueError: invalid literal for int() with base 10: '/usr/jdk64/jdk1.7.0_45/bin/java\n'
> >>> 
> 
> 
> 
> This log is from Yarn Spark execution:
>  
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/09/04 12:53:19 INFO SecurityManager: Changing view acls to: yarn,root
> 14/09/04 12:53:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root)
> 14/09/04 12:53:20 INFO Slf4jLogger: Slf4jLogger started
> 14/09/04 12:53:20 INFO Remoting: Starting remoting
> 14/09/04 12:53:20 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619]
> 14/09/04 12:53:20 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619]
> 14/09/04 12:53:20 INFO RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/10.193.1.72:8030
> 14/09/04 12:53:21 INFO ExecutorLauncher: ApplicationAttemptId: appattempt_1409805761292_0005_000001
> 14/09/04 12:53:21 INFO ExecutorLauncher: Registering the ApplicationMaster
> 14/09/04 12:53:21 INFO ExecutorLauncher: Waiting for Spark driver to be reachable.
> 14/09/04 12:53:21 INFO ExecutorLauncher: Driver now available: HDOP-B.AGT:45747
> 14/09/04 12:53:21 INFO ExecutorLauncher: Listen to driver: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler
> 14/09/04 12:53:21 INFO ExecutorLauncher: Allocating 3 executors.
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Will Allocate 3 executor containers, each with 2432 memory
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:2432, vCores:1>
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:2432, vCores:1>
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:2432, vCores:1>
> 14/09/04 12:53:21 INFO AMRMClientImpl: Received new token for : HDOP-M.AGT:45454
> 14/09/04 12:53:21 INFO AMRMClientImpl: Received new token for : HDOP-N1.AGT:45454
> 14/09/04 12:53:21 INFO RackResolver: Resolved HDOP-N1.AGT to /default-rack
> 14/09/04 12:53:21 INFO RackResolver: Resolved HDOP-M.AGT to /default-rack
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000003 for on host HDOP-N1.AGT
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-N1.AGT
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000002 for on host HDOP-M.AGT
> 14/09/04 12:53:21 INFO ExecutorRunnable: Starting Executor Container
> 14/09/04 12:53:21 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-M.AGT
> 14/09/04 12:53:21 INFO ExecutorRunnable: Starting Executor Container
> 14/09/04 12:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 14/09/04 12:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 14/09/04 12:53:21 INFO ExecutorRunnable: Setting up ContainerLaunchContext
> 14/09/04 12:53:21 INFO ExecutorRunnable: Setting up ContainerLaunchContext
> 14/09/04 12:53:21 INFO ExecutorRunnable: Preparing Local resources
> 14/09/04 12:53:21 INFO ExecutorRunnable: Preparing Local resources
> 14/09/04 12:53:21 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
> 14/09/04 12:53:21 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
> 14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 1, HDOP-N1.AGT, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 2, HDOP-M.AGT, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-N1.AGT:45454
> 14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-M.AGT:45454
> 14/09/04 12:53:22 INFO AMRMClientImpl: Received new token for : HDOP-N4.AGT:45454
> 14/09/04 12:53:22 INFO RackResolver: Resolved HDOP-N4.AGT to /default-rack
> 14/09/04 12:53:22 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000004 for on host HDOP-N4.AGT
> 14/09/04 12:53:22 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-N4.AGT
> 14/09/04 12:53:22 INFO ExecutorRunnable: Starting Executor Container
> 14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 14/09/04 12:53:22 INFO ExecutorRunnable: Setting up ContainerLaunchContext
> 14/09/04 12:53:22 INFO ExecutorRunnable: Preparing Local resources
> 14/09/04 12:53:22 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
> 14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 3, HDOP-N4.AGT, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-N4.AGT:45454
> 14/09/04 12:53:22 INFO ExecutorLauncher: All executors have launched.
> 14/09/04 12:53:22 INFO ExecutorLauncher: Started progress reporter thread - sleep time : 5000
> 14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -> [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -> [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -> [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -> [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -> [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:54:02 INFO ExecutorLauncher: finish ApplicationMaster with SUCCEEDED
> 14/09/04 12:54:02 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
> 14/09/04 12:54:02 INFO ExecutorLauncher: Exited
> 
> 
> 
> 
> Exception still occur:
> 
> 
>   [root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn  --num-executors 3  --driver-memory 4g --executor-memory 2g --executor-cores 1   examples/src/main/python/pi.py   1000
> /usr/jdk64/jdk1.7.0_45/bin/java
> ::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf
> -XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g
> 14/09/04 12:53:11 INFO spark.SecurityManager: Changing view acls to: root
> 14/09/04 12:53:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
> 14/09/04 12:53:12 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/09/04 12:53:12 INFO Remoting: Starting remoting
> 14/09/04 12:53:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:45747]
> 14/09/04 12:53:12 INFO spark.SparkEnv: Registering MapOutputTracker
> 14/09/04 12:53:12 INFO spark.SparkEnv: Registering BlockManagerMaster
> 14/09/04 12:53:12 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140904125312-c7ea
> 14/09/04 12:53:12 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.
> 14/09/04 12:53:12 INFO network.ConnectionManager: Bound socket to port 37363 with id = ConnectionManagerId(HDOP-B.AGT,37363)
> 14/09/04 12:53:12 INFO storage.BlockManagerMaster: Trying to register BlockManager
> 14/09/04 12:53:12 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:37363 with 2.3 GB RAM
> 14/09/04 12:53:12 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/09/04 12:53:12 INFO spark.HttpServer: Starting HTTP Server
> 14/09/04 12:53:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/09/04 12:53:12 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:33547
> 14/09/04 12:53:12 INFO broadcast.HttpBroadcast: Broadcast server started at http://10.193.1.76:33547
> 14/09/04 12:53:12 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-054f4eda-b93b-47d3-87d5-c40e81fc1fe8
> 14/09/04 12:53:12 INFO spark.HttpServer: Starting HTTP Server
> 14/09/04 12:53:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/09/04 12:53:12 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:54594
> 14/09/04 12:53:13 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/09/04 12:53:13 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
> 14/09/04 12:53:13 INFO ui.SparkUI: Started SparkUI at http://HDOP-B.AGT:4040
> 14/09/04 12:53:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> --args is deprecated. Use --arg instead.
> 14/09/04 12:53:14 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/10.193.1.72:8050
> 14/09/04 12:53:14 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6
> 14/09/04 12:53:14 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
>       queueApplicationCount = 0, queueChildQueueCount = 0
> 14/09/04 12:53:14 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824
> 14/09/04 12:53:14 INFO yarn.Client: Preparing Local resources
> 14/09/04 12:53:15 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
> 14/09/04 12:53:17 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409805761292_0005/pi.py
> 14/09/04 12:53:17 INFO yarn.Client: Setting up the launch environment
> 14/09/04 12:53:17 INFO yarn.Client: Setting up container launch context
> 14/09/04 12:53:17 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-2b59c845-3de2-4c3d-a352-1379ecade281\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -Dspark.app.name=\"PythonPi\", -Dspark.fileserver.uri=\"http://10.193.1.76:54594\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"45747\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"http://10.193.1.76:33547\",  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null,  --args  'HDOP-B.AGT:45747' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 14/09/04 12:53:17 INFO yarn.Client: Submitting application to ASM
> 14/09/04 12:53:17 INFO impl.YarnClientImpl: Submitted application application_1409805761292_0005
> 14/09/04 12:53:17 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
> 	 appMasterRpcPort: -1
> 	 appStartTime: 1409806397305
> 	 yarnAppState: ACCEPTED
> 
> 14/09/04 12:53:18 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
> 	 appMasterRpcPort: -1
> 	 appStartTime: 1409806397305
> 	 yarnAppState: ACCEPTED
> 
> 14/09/04 12:53:19 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
> 	 appMasterRpcPort: -1
> 	 appStartTime: 1409806397305
> 	 yarnAppState: ACCEPTED
> 
> 14/09/04 12:53:20 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
> 	 appMasterRpcPort: -1
> 	 appStartTime: 1409806397305
> 	 yarnAppState: ACCEPTED
> 
> 14/09/04 12:53:21 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
> 	 appMasterRpcPort: 0
> 	 appStartTime: 1409806397305
> 	 yarnAppState: RUNNING
> 
> 14/09/04 12:53:23 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done
> 14/09/04 12:53:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N1.AGT:40024/user/Executor#2065794895] with ID 1
> 14/09/04 12:53:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-N1.AGT:34857 with 1178.1 MB RAM
> 14/09/04 12:53:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N4.AGT:49234/user/Executor#820272849] with ID 3
> 14/09/04 12:53:27 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-M.AGT:38124/user/Executor#715249825] with ID 2
> 14/09/04 12:53:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-N4.AGT:43365 with 1178.1 MB RAM
> 14/09/04 12:53:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-M.AGT:45711 with 1178.1 MB RAM
> 14/09/04 12:53:55 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Missing parents: List()
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
> 14/09/04 12:53:55 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
> 14/09/04 12:53:55 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 5 ms
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 2 ms
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 2 ms
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 5 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
> 
> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 4 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 5 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 1]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 5 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 5 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 2]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 6 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 5 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 3]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 7 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 4 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:2)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 4]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 8 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 3 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:1)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 5]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 9 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 4 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:3)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 6]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 10 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 3 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:0)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 7]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 11 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 3 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:2)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 8]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 12 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 4 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 10 (task 0.0:3)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 9]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 13 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 3 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:1)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 10]
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 14 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 4 ms
> 14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:0)
> 14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 11]
> 14/09/04 12:53:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 15 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
> 14/09/04 12:53:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 4 ms
> 14/09/04 12:53:57 WARN scheduler.TaskSetManager: Lost TID 12 (task 0.0:2)
> 14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 12]
> 14/09/04 12:53:57 ERROR scheduler.TaskSetManager: Task 0.0:2 failed 4 times; aborting job
> 14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 13]
> 14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
> 14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
> 14/09/04 12:53:57 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
> Traceback (most recent call last):
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in <module>
>     count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce
>     vals = self.mapPartitions(func).collect()
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect
>     bytesInJava = self._jrdd.collect().iterator()
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
> py4j.protocol.Py4JJavaError14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
>  [duplicate 14]
> 14/09/04 12:53:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException
> org.apache.spark.TaskKilledException
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> : An error occurred while calling o24.collect.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:2 failed 4 times, most recent failure: Exception failure in TID 12 on host HDOP-M.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>     for obj in iterator:
>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>     for item in iterator:
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
> SystemError: unknown opcode
> 
>         org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>         org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>         org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>         org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>         org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>         org.apache.spark.scheduler.Task.run(Task.scala:51)
>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:744)
> Driver stacktrace:
> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)
> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
> 	at scala.Option.foreach(Option.scala:236)
> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)
> 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> 	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> 14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
> 
> 
> 
> What other procedure can be done for fixing the problem.
> 
> Thanks
> Oleg.
>  
> 
> 
>> On Thu, Sep 4, 2014 at 5:36 AM, Andrew Or <andrew@databricks.com> wrote:
>> Hi Oleg,
>> 
>> Your configuration looks alright to me. I haven't seen an "unknown opcode" System.error before in PySpark. This usually means you have corrupted .pyc files lying around (ones that belonged to an old python version, perhaps). What python version are you using? Are all your nodes running the same version of python? What happens if you just run bin/pyspark with the same command line arguments, and then do an "sc.parallelize(range(10)).count()", does it still fail?
>> 
>> Andrew
>> 
>> 
>> 2014-09-02 23:42 GMT-07:00 Oleg Ruchovets <oruchovets@gmail.com>:
>>> Hi I changed master to yarn but execution failed with exception again. I am using PySpark.
>>> 
>>> [root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn  --num-executors 3  --driver-memory 4g --executor-memory 2g --executor-cores 1   examples/src/main/python/pi.py   1000
>>> /usr/jdk64/jdk1.7.0_45/bin/java
>>> ::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf
>>> -XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g
>>> 14/09/03 14:35:11 INFO spark.SecurityManager: Changing view acls to: root
>>> 14/09/03 14:35:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
>>> 14/09/03 14:35:11 INFO slf4j.Slf4jLogger: Slf4jLogger started
>>> 14/09/03 14:35:11 INFO Remoting: Starting remoting
>>> 14/09/03 14:35:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:51707]
>>> 14/09/03 14:35:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:51707]
>>> 14/09/03 14:35:12 INFO spark.SparkEnv: Registering MapOutputTracker
>>> 14/09/03 14:35:12 INFO spark.SparkEnv: Registering BlockManagerMaster
>>> 14/09/03 14:35:12 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140903143512-5aab
>>> 14/09/03 14:35:12 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.
>>> 14/09/03 14:35:12 INFO network.ConnectionManager: Bound socket to port 53216 with id = ConnectionManagerId(HDOP-B.AGT,53216)
>>> 14/09/03 14:35:12 INFO storage.BlockManagerMaster: Trying to register BlockManager
>>> 14/09/03 14:35:12 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:53216 with 2.3 GB RAM
>>> 14/09/03 14:35:12 INFO storage.BlockManagerMaster: Registered BlockManager
>>> 14/09/03 14:35:12 INFO spark.HttpServer: Starting HTTP Server
>>> 14/09/03 14:35:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>> 14/09/03 14:35:12 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:50624
>>> 14/09/03 14:35:12 INFO broadcast.HttpBroadcast: Broadcast server started at http://10.193.1.76:50624
>>> 14/09/03 14:35:12 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-fd7fdcb2-f45d-430f-95fa-afbc4f329b43
>>> 14/09/03 14:35:12 INFO spark.HttpServer: Starting HTTP Server
>>> 14/09/03 14:35:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>> 14/09/03 14:35:12 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:41773
>>> 14/09/03 14:35:13 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>> 14/09/03 14:35:13 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
>>> 14/09/03 14:35:13 INFO ui.SparkUI: Started SparkUI at http://HDOP-B.AGT:4040
>>> 14/09/03 14:35:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>> --args is deprecated. Use --arg instead.
>>> 14/09/03 14:35:14 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/10.193.1.72:8050
>>> 14/09/03 14:35:14 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6
>>> 14/09/03 14:35:14 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
>>>       queueApplicationCount = 0, queueChildQueueCount = 0
>>> 14/09/03 14:35:14 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824
>>> 14/09/03 14:35:14 INFO yarn.Client: Preparing Local resources
>>> 14/09/03 14:35:14 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0036/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
>>> 14/09/03 14:35:16 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0036/pi.py
>>> 14/09/03 14:35:16 INFO yarn.Client: Setting up the launch environment
>>> 14/09/03 14:35:16 INFO yarn.Client: Setting up container launch context
>>> 14/09/03 14:35:16 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-98b7d323-2faf-419a-a88d-1a0c549dc5d4\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -Dspark.app.name=\"PythonPi\", -Dspark.fileserver.uri=\"http://10.193.1.76:41773\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"51707\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"http://10.193.1.76:50624\",  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null,  --args  'HDOP-B.AGT:51707' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>>> 14/09/03 14:35:16 INFO yarn.Client: Submitting application to ASM
>>> 14/09/03 14:35:16 INFO impl.YarnClientImpl: Submitted application application_1409559972905_0036
>>> 14/09/03 14:35:16 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:17 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:18 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:19 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:20 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:21 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: -1
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: ACCEPTED
>>> 
>>> 14/09/03 14:35:22 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>> 	 appMasterRpcPort: 0
>>> 	 appStartTime: 1409726116517
>>> 	 yarnAppState: RUNNING
>>> 
>>> 14/09/03 14:35:24 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done
>>> 14/09/03 14:35:25 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:58976/user/Executor#-1831707618] with ID 1
>>> 14/09/03 14:35:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:44142 with 1178.1 MB RAM
>>> 14/09/03 14:35:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N1.AGT:45140/user/Executor#875812337] with ID 2
>>> 14/09/03 14:35:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-N1.AGT:48513 with 1178.1 MB RAM
>>> 14/09/03 14:35:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N3.AGT:45380/user/Executor#1559437246] with ID 3
>>> 14/09/03 14:35:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-N3.AGT:46616 with 1178.1 MB RAM
>>> 14/09/03 14:35:56 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Parents of final stage: List()
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Missing parents: List()
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
>>> 14/09/03 14:35:56 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
>>> 14/09/03 14:35:56 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks
>>> 14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 9 ms
>>> 14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 5 ms
>>> 14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 5 ms
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
>>> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>> 
>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 	at java.lang.Thread.run(Thread.java:744)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 4 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
>>> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>> 
>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 	at java.lang.Thread.run(Thread.java:744)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 5 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 3 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 1]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 6 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:2)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 1]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 7 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 2]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 8 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:0)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 3]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 9 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:3)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 2]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 10 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:2)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 4]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 11 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:0)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 3]
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 12 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms
>>> 14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:1)
>>> 14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 5]
>>> 14/09/03 14:35:58 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 13 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)
>>> 14/09/03 14:35:58 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 3 ms
>>> 14/09/03 14:35:58 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:2)
>>> 14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 4]
>>> 14/09/03 14:35:58 ERROR scheduler.TaskSetManager: Task 0.0:2 failed 4 times; aborting job
>>> 14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
>>> 14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
>>> 14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 6]
>>> 14/09/03 14:35:58 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
>>> Traceback (most recent call last):
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in <module>
>>>     count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce
>>>     vals = self.mapPartitions(func).collect()
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect
>>>     bytesInJava = self._jrdd.collect().iterator()
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
>>> py4j.protocol.Py4JJavaError14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>>  [duplicate 7]
>>> : An error occurred while calling o24.collect.
>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:2 failed 4 times, most recent failure: Exception failure in TID 11 on host HDOP-N1.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>     for obj in iterator:
>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>     for item in iterator:
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>> SystemError: unknown opcode
>>> 
>>>         org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>>         org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>>         org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>>         org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>         org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>>         org.apache.spark.scheduler.Task.run(Task.scala:51)
>>>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>         java.lang.Thread.run(Thread.java:744)
>>> Driver stacktrace:
>>> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)
>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)
>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)
>>> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)
>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
>>> 	at scala.Option.foreach(Option.scala:236)
>>> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)
>>> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)
>>> 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
>>> 	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
>>> 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
>>> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>>> 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>> 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> 
>>> 14/09/03 14:35:58 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException
>>> org.apache.spark.TaskKilledException
>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> 	at java.lang.Thread.run(Thread.java:744)
>>> 14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
>>> 
>>>    
>>> 
>>> 
>>>> On Wed, Sep 3, 2014 at 1:53 PM, Oleg Ruchovets <oruchovets@gmail.com> wrote:
>>>> Hello Sandy , I changed to using yarn master but still got the exceptions:
>>>> 
>>>> What is the procedure to execute pyspark on yarn? is it required only to attached the command , or it is required to start spark processes also?
>>>> 
>>>> 
>>>> 
>>>> 
>>>> [root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn://HDOP-N1.AGT:8032 --num-executors 3  --driver-memory 4g --executor-memory 2g --executor-cores 1   examples/src/main/python/pi.py   1000
>>>> /usr/jdk64/jdk1.7.0_45/bin/java
>>>> ::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf
>>>> -XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g
>>>> 14/09/03 13:48:48 INFO spark.SecurityManager: Changing view acls to: root
>>>> 14/09/03 13:48:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
>>>> 14/09/03 13:48:49 INFO slf4j.Slf4jLogger: Slf4jLogger started
>>>> 14/09/03 13:48:49 INFO Remoting: Starting remoting
>>>> 14/09/03 13:48:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:34424]
>>>> 14/09/03 13:48:49 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:34424]
>>>> 14/09/03 13:48:49 INFO spark.SparkEnv: Registering MapOutputTracker
>>>> 14/09/03 13:48:49 INFO spark.SparkEnv: Registering BlockManagerMaster
>>>> 14/09/03 13:48:49 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140903134849-231c
>>>> 14/09/03 13:48:49 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.
>>>> 14/09/03 13:48:49 INFO network.ConnectionManager: Bound socket to port 60647 with id = ConnectionManagerId(HDOP-B.AGT,60647)
>>>> 14/09/03 13:48:49 INFO storage.BlockManagerMaster: Trying to register BlockManager
>>>> 14/09/03 13:48:49 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:60647 with 2.3 GB RAM
>>>> 14/09/03 13:48:49 INFO storage.BlockManagerMaster: Registered BlockManager
>>>> 14/09/03 13:48:49 INFO spark.HttpServer: Starting HTTP Server
>>>> 14/09/03 13:48:49 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>> 14/09/03 13:48:49 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:56549
>>>> 14/09/03 13:48:49 INFO broadcast.HttpBroadcast: Broadcast server started at http://10.193.1.76:56549
>>>> 14/09/03 13:48:49 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-90af1222-9ea8-4dd8-887a-343d09d44333
>>>> 14/09/03 13:48:49 INFO spark.HttpServer: Starting HTTP Server
>>>> 14/09/03 13:48:49 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>> 14/09/03 13:48:49 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:36512
>>>> 14/09/03 13:48:50 INFO server.Server: jetty-8.y.z-SNAPSHOT
>>>> 14/09/03 13:48:50 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
>>>> 14/09/03 13:48:50 INFO ui.SparkUI: Started SparkUI at http://HDOP-B.AGT:4040
>>>> 14/09/03 13:48:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>> --args is deprecated. Use --arg instead.
>>>> 14/09/03 13:48:51 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/10.193.1.72:8050
>>>> 14/09/03 13:48:51 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6
>>>> 14/09/03 13:48:51 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
>>>>       queueApplicationCount = 0, queueChildQueueCount = 0
>>>> 14/09/03 13:48:51 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824
>>>> 14/09/03 13:48:51 INFO yarn.Client: Preparing Local resources
>>>> 14/09/03 13:48:51 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0033/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
>>>> 14/09/03 13:48:53 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0033/pi.py
>>>> 14/09/03 13:48:53 INFO yarn.Client: Setting up the launch environment
>>>> 14/09/03 13:48:53 INFO yarn.Client: Setting up container launch context
>>>> 14/09/03 13:48:53 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-bdabb882-a2e0-46b6-8e87-90cc6e359d84\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -Dspark.app.name=\"PythonPi\", -Dspark.fileserver.uri=\"http://10.193.1.76:36512\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"34424\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"http://10.193.1.76:56549\",  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null,  --args  'HDOP-B.AGT:34424' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>>>> 14/09/03 13:48:53 INFO yarn.Client: Submitting application to ASM
>>>> 14/09/03 13:48:53 INFO impl.YarnClientImpl: Submitted application application_1409559972905_0033
>>>> 14/09/03 13:48:53 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: -1
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: ACCEPTED
>>>> 
>>>> 14/09/03 13:48:54 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: -1
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: ACCEPTED
>>>> 
>>>> 14/09/03 13:48:55 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: -1
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: ACCEPTED
>>>> 
>>>> 14/09/03 13:48:56 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: -1
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: ACCEPTED
>>>> 
>>>> 14/09/03 13:48:57 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: -1
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: ACCEPTED
>>>> 
>>>> 14/09/03 13:48:58 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
>>>> 	 appMasterRpcPort: 0
>>>> 	 appStartTime: 1409723333584
>>>> 	 yarnAppState: RUNNING
>>>> 
>>>> 14/09/03 13:49:00 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done
>>>> 14/09/03 13:49:01 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:57078/user/Executor#1595833626] with ID 1
>>>> 14/09/03 13:49:02 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:54579 with 1178.1 MB RAM
>>>> 14/09/03 13:49:03 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N4.AGT:43121/user/Executor#-1266627304] with ID 2
>>>> 14/09/03 13:49:03 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N2.AGT:36952/user/Executor#1003961369] with ID 3
>>>> 14/09/03 13:49:04 INFO storage.BlockManagerInfo: Registering block manager HDOP-N4.AGT:56891 with 1178.1 MB RAM
>>>> 14/09/03 13:49:04 INFO storage.BlockManagerInfo: Registering block manager HDOP-N2.AGT:42381 with 1178.1 MB RAM
>>>> 14/09/03 13:49:33 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Parents of final stage: List()
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Missing parents: List()
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
>>>> 14/09/03 13:49:33 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
>>>> 14/09/03 13:49:33 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 5 ms
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 5 ms
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 5 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
>>>> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>> 
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>>> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 4 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
>>>> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>> 
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>>> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 5 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
>>>> org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>> 
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>>> 	at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>>> 	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>>> 	at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 6 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 1]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 7 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:2)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 1]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 8 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:1)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 1]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 9 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:0)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 2]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 10 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 3 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:3)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 2]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 11 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:2)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 2]
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 12 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms
>>>> 14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:1)
>>>> 14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 3]
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 13 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms
>>>> 14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 10 (task 0.0:0)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 3]
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 14 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms
>>>> 14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:3)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 3]
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 15 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 3 ms
>>>> 14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 13 (task 0.0:1)
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 4]
>>>> 14/09/03 13:49:35 ERROR scheduler.TaskSetManager: Task 0.0:1 failed 4 times; aborting job
>>>> 14/09/03 13:49:35 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
>>>> 14/09/03 13:49:35 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
>>>> 14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>>  [duplicate 4]
>>>> 14/09/03 13:49:35 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
>>>> Traceback (most recent call last):
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in <module>
>>>>     count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce
>>>>     vals = self.mapPartitions(func).collect()
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect
>>>>     bytesInJava = self._jrdd.collect().iterator()
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
>>>> py4j.protocol.Py4JJavaError: An error occurred while calling o24.collect.
>>>> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:1 failed 4 times, most recent failure: Exception failure in TID 13 on host HDOP-N2.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
>>>>     serializer.dump_stream(func(split_index, iterator), outfile)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
>>>>     self.serializer.dump_stream(self._batched(iterator), stream)
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
>>>>     for obj in iterator:
>>>>   File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
>>>>     for item in iterator:
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
>>>> SystemError: unknown opcode
>>>> 
>>>>         org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
>>>>         org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:145)
>>>>         org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
>>>>         org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>>         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>>         org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>>>>         org.apache.spark.scheduler.Task.run(Task.scala:51)
>>>>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>>>>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>         java.lang.Thread.run(Thread.java:744)
>>>> Driver stacktrace:
>>>> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)
>>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)
>>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)
>>>> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>>> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>>> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)
>>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
>>>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
>>>> 	at scala.Option.foreach(Option.scala:236)
>>>> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)
>>>> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)
>>>> 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
>>>> 	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
>>>> 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
>>>> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>>>> 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>>> 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 
>>>> 14/09/03 13:49:35 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException
>>>> org.apache.spark.TaskKilledException
>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>> 
>>>> 
>>>>> On Wed, Sep 3, 2014 at 1:40 PM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>>>>> Hi Oleg. To run on YARN, simply set master to "yarn".  The YARN configuration, located in a yarn-site.xml, determines where to look for the YARN ResourceManager.
>>>>> 
>>>>> PROCESS_LOCAL is orthogonal to the choice of cluster resource manager. A task is considered PROCESS_LOCAL when the executor it's running in happens to have the data it's processing cached.
>>>>> 
>>>>> If you're looking to get familiar with the kind of confusing web of terminology, this blog post might be helpful: 
>>>>> http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/
>>>>> 
>>>>> -Sandy
>>>>> 
>>>>> 
>>>>>> On Tue, Sep 2, 2014 at 9:51 PM, Oleg Ruchovets <oruchovets@gmail.com> wrote:
>>>>>> Hi , 
>>>>>>   I change my command to :
>>>>>>   ./bin/spark-submit --master spark://HDOP-B.AGT:7077 --num-executors 3  --driver-memory 4g --executor-memory 2g --executor-cores 1   examples/src/main/python/pi.py   1000
>>>>>> and it fixed the problem.
>>>>>> 
>>>>>> I still have couple of questions: 
>>>>>>    PROCESS_LOCAL is not Yarn execution , right? how should I configure the running on yarn? Should I exeture start-all script on all machine or only one?  Where is the UI / LOGS of spark execution?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>    
>>>>>> 
>>>>>> 152	 152	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:14	 0.2 s			
>>>>>> 0	0	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:09	0.9 s	 39 ms		
>>>>>> 2	 2	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:09	 0.9 s	 39 ms		
>>>>>> 3	3	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:09	0.9 s	 39 ms	1 ms	
>>>>>> 4	 4	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:09	 0.8 s	 39 ms	 2 ms	
>>>>>> 5	5	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:09	0.8 s	 39 ms	1 ms	
>>>>>> 6	 6	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:09	 0.8 s		 1 ms	
>>>>>> 7	7	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:09	0.9 s			
>>>>>> 8	 8	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:10	 0.3 s			
>>>>>> 9	9	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:10	0.4 s			
>>>>>> 10	 10	 SUCCESS	 PROCESS_LOCAL	 HDOP-B.AGT	 2014/09/03 12:35:10	 0.3 s		 1 ms	
>>>>>> 11	11	 SUCCESS	PROCESS_LOCAL	HDOP-B.AGT	2014/09/03 12:35:10	0.3 s	
>>>>>> 
>>>>>> 
>>>>>>> On Wed, Sep 3, 2014 at 12:19 PM, Oleg Ruchovets <oruchovets@gmail.com> wrote:
>>>>>>> Hi Andrew.
>>>>>>>    what should I do to set master on yarn, can you please pointing me on command or documentation how to do it?
>>>>>>> 
>>>>>>> 
>>>>>>> I am doing the following:
>>>>>>>    executed start-all.sh
>>>>>>>    [root@HDOP-B sbin]# ./start-all.sh 
>>>>>>> starting org.apache.spark.deploy.master.Master, logging to /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-HDOP-B.AGT.out
>>>>>>> localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
>>>>>>> localhost: starting org.apache.spark.deploy.worker.Worker, logging to /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-HDOP-B.AGT.out
>>>>>>> 
>>>>>>> 
>>>>>>> after execute the command:
>>>>>>>     ./bin/spark-submit --master spark://HDOP-B.AGT:7077 examples/src/main/python/pi.py 1000
>>>>>>> 
>>>>>>> 
>>>>>>> the result is the following:
>>>>>>> 
>>>>>>>    /usr/jdk64/jdk1.7.0_45/bin/java
>>>>>>> ::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
>>>>>>> -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m
>>>>>>> 14/09/03 12:10:06 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
>>>>>>> 14/09/03 12:10:06 INFO SecurityManager: Changing view acls to: root
>>>>>>> 14/09/03 12:10:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
>>>>>>> 14/09/03 12:10:07 INFO Slf4jLogger: Slf4jLogger started
>>>>>>> 14/09/03 12:10:07 INFO Remoting: Starting remoting
>>>>>>> 14/09/03 12:10:07 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:38944]
>>>>>>> 14/09/03 12:10:07 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:38944]
>>>>>>> 14/09/03 12:10:07 INFO SparkEnv: Registering MapOutputTracker
>>>>>>> 14/09/03 12:10:07 INFO SparkEnv: Registering BlockManagerMaster
>>>>>>> 14/09/03 12:10:08 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140903121008-cf09
>>>>>>> 14/09/03 12:10:08 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.
>>>>>>> 14/09/03 12:10:08 INFO ConnectionManager: Bound socket to port 45041 with id = ConnectionManagerId(HDOP-B.AGT,45041)
>>>>>>> 14/09/03 12:10:08 INFO BlockManagerMaster: Trying to register BlockManager
>>>>>>> 14/09/03 12:10:08 INFO BlockManagerInfo: Registering block manager HDOP-B.AGT:45041 with 294.9 MB RAM
>>>>>>> 14/09/03 12:10:08 INFO BlockManagerMaster: Registered BlockManager
>>>>>>> 14/09/03 12:10:08 INFO HttpServer: Starting HTTP Server
>>>>>>> 14/09/03 12:10:08 INFO HttpBroadcast: Broadcast server started at http://10.193.1.76:59336
>>>>>>> 14/09/03 12:10:08 INFO HttpFileServer: HTTP File server directory is /tmp/spark-7bf5c3c3-1c02-41e8-9fb0-983e175dd45c
>>>>>>> 14/09/03 12:10:08 INFO HttpServer: Starting HTTP Server
>>>>>>> 14/09/03 12:10:08 INFO SparkUI: Started SparkUI at http://HDOP-B.AGT:4040
>>>>>>> 14/09/03 12:10:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>>> 14/09/03 12:10:09 INFO Utils: Copying /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to /tmp/spark-4e252376-70cb-4171-bf2c-d804524e816c/pi.py
>>>>>>> 14/09/03 12:10:09 INFO SparkContext: Added file file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py at http://10.193.1.76:45893/files/pi.py with timestamp 1409717409277
>>>>>>> 14/09/03 12:10:09 INFO AppClient$ClientActor: Connecting to master spark://HDOP-B.AGT:7077...
>>>>>>> 14/09/03 12:10:09 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140903121009-0000
>>>>>>> 14/09/03 12:10:09 INFO AppClient$ClientActor: Executor added: app-20140903121009-0000/0 on worker-20140903120712-HDOP-B.AGT-51161 (HDOP-B.AGT:51161) with 8 cores
>>>>>>> 14/09/03 12:10:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140903121009-0000/0 on hostPort HDOP-B.AGT:51161 with 8 cores, 512.0 MB RAM
>>>>>>> 14/09/03 12:10:09 INFO AppClient$ClientActor: Executor updated: app-20140903121009-0000/0 is now RUNNING
>>>>>>> 14/09/03 12:10:12 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:38143/user/Executor#1295757828] with ID 0
>>>>>>> 14/09/03 12:10:12 INFO BlockManagerInfo: Registering block manager HDOP-B.AGT:38670 with 294.9 MB RAM
>>>>>>> Traceback (most recent call last):
>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in <module>
>>>>>>>     count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 271, in parallelize
>>>>>>>     jrdd = readRDDFromFile(self._jsc, tempFile.name, numSlices)
>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
>>>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.
>>>>>>> : java.lang.OutOfMemoryError: Java heap space
>>>>>>> 	at org.apache.spark.api.python.PythonRDD$.readRDDFromFile(PythonRDD.scala:279)
>>>>>>> 	at org.apache.spark.api.python.PythonRDD.readRDDFromFile(PythonRDD.scala)
>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> 	at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
>>>>>>> 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
>>>>>>> 	at py4j.Gateway.invoke(Gateway.java:259)
>>>>>>> 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>> 	at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>> 	at py4j.GatewayConnection.run(GatewayConnection.java:207)
>>>>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> What should I do to fix the issue 
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Oleg.
>>>>>>> 
>>>>>>> 
>>>>>>>> On Tue, Sep 2, 2014 at 10:32 PM, Andrew Or <andrew@databricks.com> wrote:
>>>>>>>> Hi Oleg,
>>>>>>>> 
>>>>>>>> If you are running Spark on a yarn cluster, you should set --master to yarn. By default this runs in client mode, which redirects all output of your application to your console. This is failing because it is trying to connect to a standalone master that you probably did not start. I am somewhat puzzled as to how you ran into an OOM from this configuration, however. Does this problem still occur if you set the correct master?
>>>>>>>> 
>>>>>>>> -Andrew
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 2014-09-02 2:42 GMT-07:00 Oleg Ruchovets <oruchovets@gmail.com>:
>>>>>>>> 
>>>>>>>>> Hi , 
>>>>>>>>>    I've installed pyspark on hpd hortonworks cluster. 
>>>>>>>>>   Executing pi example:
>>>>>>>>> 
>>>>>>>>> command:
>>>>>>>>>        spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master spark://10.193.1.71:7077   examples/src/main/python/pi.py   1000
>>>>>>>>> 
>>>>>>>>> exception:
>>>>>>>>> 
>>>>>>>>>     14/09/02 17:34:02 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
>>>>>>>>> 14/09/02 17:34:02 INFO SecurityManager: Changing view acls to: root
>>>>>>>>> 14/09/02 17:34:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
>>>>>>>>> 14/09/02 17:34:02 INFO Slf4jLogger: Slf4jLogger started
>>>>>>>>> 14/09/02 17:34:02 INFO Remoting: Starting remoting
>>>>>>>>> 14/09/02 17:34:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-M.AGT:41059]
>>>>>>>>> 14/09/02 17:34:03 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-M.AGT:41059]
>>>>>>>>> 14/09/02 17:34:03 INFO SparkEnv: Registering MapOutputTracker
>>>>>>>>> 14/09/02 17:34:03 INFO SparkEnv: Registering BlockManagerMaster
>>>>>>>>> 14/09/02 17:34:03 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140902173403-cda8
>>>>>>>>> 14/09/02 17:34:03 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.
>>>>>>>>> 14/09/02 17:34:03 INFO ConnectionManager: Bound socket to port 34931 with id = ConnectionManagerId(HDOP-M.AGT,34931)
>>>>>>>>> 14/09/02 17:34:03 INFO BlockManagerMaster: Trying to register BlockManager
>>>>>>>>> 14/09/02 17:34:03 INFO BlockManagerInfo: Registering block manager HDOP-M.AGT:34931 with 294.9 MB RAM
>>>>>>>>> 14/09/02 17:34:03 INFO BlockManagerMaster: Registered BlockManager
>>>>>>>>> 14/09/02 17:34:03 INFO HttpServer: Starting HTTP Server
>>>>>>>>> 14/09/02 17:34:03 INFO HttpBroadcast: Broadcast server started at http://10.193.1.71:54341
>>>>>>>>> 14/09/02 17:34:03 INFO HttpFileServer: HTTP File server directory is /tmp/spark-77c7a7dc-181e-4069-a014-8103a6a6330a
>>>>>>>>> 14/09/02 17:34:03 INFO HttpServer: Starting HTTP Server
>>>>>>>>> 14/09/02 17:34:04 INFO SparkUI: Started SparkUI at http://HDOP-M.AGT:4040
>>>>>>>>> 14/09/02 17:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>>>>>>>> 14/09/02 17:34:04 INFO Utils: Copying /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to /tmp/spark-f2e0cc0f-59cb-4f6c-9d48-f16205a40c7e/pi.py
>>>>>>>>> 14/09/02 17:34:04 INFO SparkContext: Added file file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py at http://10.193.1.71:52938/files/pi.py with timestamp 1409650444941
>>>>>>>>> 14/09/02 17:34:05 INFO AppClient$ClientActor: Connecting to master spark://10.193.1.71:7077...
>>>>>>>>> 14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:25 INFO AppClient$ClientActor: Connecting to master spark://10.193.1.71:7077...
>>>>>>>>> 14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> 14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@10.193.1.71:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@10.193.1.71:7077]
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in <module>
>>>>>>>>>     count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
>>>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 271, in parallelize
>>>>>>>>>     jrdd = readRDDFromFile(self._jsc, tempFile.name, numSlices)
>>>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>>>>>>>>>   File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
>>>>>>>>> py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.
>>>>>>>>> : java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>> 	at org.apache.spark.api.python.PythonRDD$.readRDDFromFile(PythonRDD.scala:279)
>>>>>>>>> 	at org.apache.spark.api.python.PythonRDD.readRDDFromFile(PythonRDD.scala)
>>>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>> 	at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>> 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
>>>>>>>>> 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
>>>>>>>>> 	at py4j.Gateway.invoke(Gateway.java:259)
>>>>>>>>> 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>>>>>>>>> 	at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>>>>>>> 	at py4j.GatewayConnection.run(GatewayConnection.java:207)
>>>>>>>>> 	at java.lang.Thread.run(Thread.java:744)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Question: 
>>>>>>>>>     how can I know spark master and port? Where is it defined?
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> Oleg.
> 

References: <CAMFcGX13DHw4zLc+pco04VnJJmH5mzKjOLzq9vSVKqcmOp5g@mail.gmail.com> <CAMJOb8‰moAt9H1vyWdyA03e5kCOSSjxmYeNuGdsK4s8rC7BA@mail.g
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Are you certain the executors are using the same python? &nbsp;What is in PYSPARK_PYTHON? &nbsp;</div><div><br></div><div>I had a similar issue that arose from using one python on the driver's path and a different one on the cluster&nbsp;</div><div><br></div><div>It can be useful to create a string that has a small python program to describe the environment on the workers and</div><div>Then eval that string in a distributed map function and collect the result.&nbsp;<br><br>----<div>Eric Friedman</div></div><div><br>On Sep 3, 2014, at 10:15 PM, Oleg Ruchovets &lt;<a href="mailto:oruchovets@gmail.com">oruchovets@gmail.com</a>&gt; wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Hi Andrew.<div><br></div><div>Problem still occur:&nbsp;<br><div style=""><br></div><div style=""><b>all machines are using python 2.7:</b><br></div><div style=""><br></div><div style="">[root@HDOP-N2 conf]# python --version</div>
<div style="">Python 2.7.7 :: Anaconda 2.0.1 (64-bit)<br></div><div style=""><br></div><div style=""><b>Executing command from bin/pyspark:</b></div><div style="">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# bin/pyspark &nbsp; &nbsp;--driver-memory 4g --executor-memory 2g --executor-cores 1 &nbsp; &nbsp; &nbsp; examples/src/main/python/pi.py &nbsp; 1000</div>
<div style=""><br></div><div style=""><br></div><div>Python 2.7.7 |Anaconda 2.0.1 (64-bit)| (default, Jun &nbsp;2 2014, 12:34:02)&nbsp;</div><div>[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2</div><div>Type "help", "copyright", "credits" or "license" for more information.</div>
<div>Anaconda is brought to you by Continuum Analytics.</div><div>Please check out: <a href="http://continuum.io/thanks">http://continuum.io/thanks</a> and <a href="https://binstar.org">https://binstar.org</a></div><div>Traceback (most recent call last):</div>
<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/shell.py", line 43, in &lt;module&gt;</div><div>&nbsp; &nbsp; sc = SparkContext(appName="PySparkShell", pyFiles=add_files)</div><div>
&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 94, in __init__</div><div>&nbsp; &nbsp; SparkContext._ensure_initialized(self, gateway=gateway)</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 190, in _ensure_initialized</div>
<div>&nbsp; &nbsp; SparkContext._gateway = gateway or launch_gateway()</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/java_gateway.py", line 51, in launch_gateway</div><div>&nbsp; &nbsp; gateway_port = int(proc.stdout.readline())</div>
<div>ValueError: invalid literal for int() with base 10: '/usr/jdk64/jdk1.7.0_45/bin/java\n'</div><div>&gt;&gt;&gt;&nbsp;</div><div><br></div><div style=""><br></div><div style=""><br></div><div style=""><b>This log is from Yarn Spark execution:</b></div>
<div style="">&nbsp;</div><div style=""><pre style="margin-top:0px;margin-bottom:0px;border:0px">SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/09/04 12:53:19 INFO SecurityManager: Changing view acls to: yarn,root
14/09/04 12:53:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root)
14/09/04 12:53:20 INFO Slf4jLogger: Slf4jLogger started
14/09/04 12:53:20 INFO Remoting: Starting remoting
14/09/04 12:53:20 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619]
14/09/04 12:53:20 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619]
14/09/04 12:53:20 INFO RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/<a href="http://10.193.1.72:8030">10.193.1.72:8030</a>
14/09/04 12:53:21 INFO ExecutorLauncher: ApplicationAttemptId: appattempt_1409805761292_0005_000001
14/09/04 12:53:21 INFO ExecutorLauncher: Registering the ApplicationMaster
14/09/04 12:53:21 INFO ExecutorLauncher: Waiting for Spark driver to be reachable.
14/09/04 12:53:21 INFO ExecutorLauncher: Driver now available: HDOP-B.AGT:45747
14/09/04 12:53:21 INFO ExecutorLauncher: Listen to driver: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler
14/09/04 12:53:21 INFO ExecutorLauncher: Allocating 3 executors.
14/09/04 12:53:21 INFO YarnAllocationHandler: Will Allocate 3 executor containers, each with 2432 memory
14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: &lt;memory:2432, vCores:1&gt;
14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: &lt;memory:2432, vCores:1&gt;
14/09/04 12:53:21 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: &lt;memory:2432, vCores:1&gt;
14/09/04 12:53:21 INFO AMRMClientImpl: Received new token for : HDOP-M.AGT:45454
14/09/04 12:53:21 INFO AMRMClientImpl: Received new token for : HDOP-N1.AGT:45454
14/09/04 12:53:21 INFO RackResolver: Resolved HDOP-N1.AGT to /default-rack
14/09/04 12:53:21 INFO RackResolver: Resolved HDOP-M.AGT to /default-rack
14/09/04 12:53:21 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000003 for on host HDOP-N1.AGT
14/09/04 12:53:21 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-N1.AGT
14/09/04 12:53:21 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000002 for on host HDOP-M.AGT
14/09/04 12:53:21 INFO ExecutorRunnable: Starting Executor Container
14/09/04 12:53:21 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-M.AGT
14/09/04 12:53:21 INFO ExecutorRunnable: Starting Executor Container
14/09/04 12:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/09/04 12:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/09/04 12:53:21 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/09/04 12:53:21 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/09/04 12:53:21 INFO ExecutorRunnable: Preparing Local resources
14/09/04 12:53:21 INFO ExecutorRunnable: Preparing Local resources
14/09/04 12:53:21 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
14/09/04 12:53:21 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 1, HDOP-N1.AGT, 1, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)
14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 2, HDOP-M.AGT, 1, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)
14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-N1.AGT:45454
14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-M.AGT:45454
14/09/04 12:53:22 INFO AMRMClientImpl: Received new token for : HDOP-N4.AGT:45454
14/09/04 12:53:22 INFO RackResolver: Resolved HDOP-N4.AGT to /default-rack
14/09/04 12:53:22 INFO YarnAllocationHandler: Launching container container_1409805761292_0005_01_000004 for on host HDOP-N4.AGT
14/09/04 12:53:22 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler,  executorHostname: HDOP-N4.AGT
14/09/04 12:53:22 INFO ExecutorRunnable: Starting Executor Container
14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/09/04 12:53:22 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/09/04 12:53:22 INFO ExecutorRunnable: Preparing Local resources
14/09/04 12:53:22 INFO ExecutorRunnable: Prepared Local resources Map(pi.py -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/pi.py" } size: 1317 timestamp: 1409806397200 type: FILE visibility: PRIVATE, __spark__.jar -&gt; resource { scheme: "hdfs" host: "HDOP-B.AGT" port: 8020 file: "/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar" } size: 121759562 timestamp: 1409806397057 type: FILE visibility: PRIVATE)
14/09/04 12:53:22 INFO ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@HDOP-B.AGT:45747/user/CoarseGrainedScheduler, 3, HDOP-N4.AGT, 1, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)
14/09/04 12:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : HDOP-N4.AGT:45454
14/09/04 12:53:22 INFO ExecutorLauncher: All executors have launched.
14/09/04 12:53:22 INFO ExecutorLauncher: Started progress reporter thread - sleep time : 5000
14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -&gt; [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -&gt; [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -&gt; [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -&gt; [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:57 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@HDOP-N2.AGT:46619] -&gt; [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:54:02 INFO ExecutorLauncher: finish ApplicationMaster with SUCCEEDED
14/09/04 12:54:02 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
14/09/04 12:54:02 INFO ExecutorLauncher: Exited</pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><b>Exception still occur:</b></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px">  [root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn  --num-executors 3  --driver-memory 4g --executor-memory 2g --executor-cores 1   examples/src/main/python/pi.py   1000
/usr/jdk64/jdk1.7.0_45/bin/java
::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf
-XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g
14/09/04 12:53:11 INFO spark.SecurityManager: Changing view acls to: root
14/09/04 12:53:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)
14/09/04 12:53:12 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/09/04 12:53:12 INFO Remoting: Starting remoting
14/09/04 12:53:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:45747]
14/09/04 12:53:12 INFO spark.SparkEnv: Registering MapOutputTracker
14/09/04 12:53:12 INFO spark.SparkEnv: Registering BlockManagerMaster
14/09/04 12:53:12 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140904125312-c7ea
14/09/04 12:53:12 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.
14/09/04 12:53:12 INFO network.ConnectionManager: Bound socket to port 37363 with id = ConnectionManagerId(HDOP-B.AGT,37363)
14/09/04 12:53:12 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/09/04 12:53:12 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:37363 with 2.3 GB RAM
14/09/04 12:53:12 INFO storage.BlockManagerMaster: Registered BlockManager
14/09/04 12:53:12 INFO spark.HttpServer: Starting HTTP Server
14/09/04 12:53:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/09/04 12:53:12 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:33547">SocketConnector@0.0.0.0:33547</a>
14/09/04 12:53:12 INFO broadcast.HttpBroadcast: Broadcast server started at <a href="http://10.193.1.76:33547">http://10.193.1.76:33547</a>
14/09/04 12:53:12 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-054f4eda-b93b-47d3-87d5-c40e81fc1fe8
14/09/04 12:53:12 INFO spark.HttpServer: Starting HTTP Server
14/09/04 12:53:12 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/09/04 12:53:12 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:54594">SocketConnector@0.0.0.0:54594</a>
14/09/04 12:53:13 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/09/04 12:53:13 INFO server.AbstractConnector: Started <a href="http://SelectChannelConnector@0.0.0.0:4040">SelectChannelConnector@0.0.0.0:4040</a>
14/09/04 12:53:13 INFO ui.SparkUI: Started SparkUI at <a href="http://HDOP-B.AGT:4040">http://HDOP-B.AGT:4040</a>
14/09/04 12:53:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
--args is deprecated. Use --arg instead.
14/09/04 12:53:14 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/<a href="http://10.193.1.72:8050">10.193.1.72:8050</a>
14/09/04 12:53:14 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6
14/09/04 12:53:14 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
      queueApplicationCount = 0, queueChildQueueCount = 0
14/09/04 12:53:14 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824
14/09/04 12:53:14 INFO yarn.Client: Preparing Local resources
14/09/04 12:53:15 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409805761292_0005/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar
14/09/04 12:53:17 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409805761292_0005/pi.py
14/09/04 12:53:17 INFO yarn.Client: Setting up the launch environment
14/09/04 12:53:17 INFO yarn.Client: Setting up container launch context
14/09/04 12:53:17 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-2b59c845-3de2-4c3d-a352-1379ecade281\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -<a href="http://Dspark.app.name">Dspark.app.name</a>=\"PythonPi\", -Dspark.fileserver.uri=\"<a href="http://10.193.1.76:54594">http://10.193.1.76:54594</a>\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"45747\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"<a href="http://10.193.1.76:33547">http://10.193.1.76:33547</a>\",  -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null,  --args  'HDOP-B.AGT:45747' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)
14/09/04 12:53:17 INFO yarn.Client: Submitting application to ASM
14/09/04 12:53:17 INFO impl.YarnClientImpl: Submitted application application_1409805761292_0005
14/09/04 12:53:17 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
	 appMasterRpcPort: -1
	 appStartTime: 1409806397305
	 yarnAppState: ACCEPTED

14/09/04 12:53:18 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
	 appMasterRpcPort: -1
	 appStartTime: 1409806397305
	 yarnAppState: ACCEPTED

14/09/04 12:53:19 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
	 appMasterRpcPort: -1
	 appStartTime: 1409806397305
	 yarnAppState: ACCEPTED

14/09/04 12:53:20 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
	 appMasterRpcPort: -1
	 appStartTime: 1409806397305
	 yarnAppState: ACCEPTED

14/09/04 12:53:21 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
	 appMasterRpcPort: 0
	 appStartTime: 1409806397305
	 yarnAppState: RUNNING

14/09/04 12:53:23 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done
14/09/04 12:53:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N1.AGT:40024/user/Executor#2065794895] with ID 1
14/09/04 12:53:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-N1.AGT:34857 with 1178.1 MB RAM
14/09/04 12:53:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N4.AGT:49234/user/Executor#820272849] with ID 3
14/09/04 12:53:27 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-M.AGT:38124/user/Executor#715249825] with ID 2
14/09/04 12:53:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-N4.AGT:43365 with 1178.1 MB RAM
14/09/04 12:53:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-M.AGT:45711 with 1178.1 MB RAM
14/09/04 12:53:55 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Missing parents: List()
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
14/09/04 12:53:55 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
14/09/04 12:53:55 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 5 ms
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 2 ms
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
14/09/04 12:53:55 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 2 ms
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 5 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode

	at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
	at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
	at org.apache.spark.scheduler.Task.run(Task.scala:51)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 4 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 5 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 1]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 5 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 5 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 2]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 6 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 5 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 3]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 7 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 4 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:2)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 4]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 8 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 3 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:1)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 5]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 9 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 4 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:3)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 6]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 10 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 3 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:0)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 7]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 11 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 3 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:2)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 8]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 12 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501135 bytes in 4 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 10 (task 0.0:3)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 9]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 13 on executor 3: HDOP-N4.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506275 bytes in 3 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:1)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 10]
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 14 on executor 1: HDOP-N1.AGT (PROCESS_LOCAL)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506275 bytes in 4 ms
14/09/04 12:53:56 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:0)
14/09/04 12:53:56 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 11]
14/09/04 12:53:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 15 on executor 2: HDOP-M.AGT (PROCESS_LOCAL)
14/09/04 12:53:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369810 bytes in 4 ms
14/09/04 12:53:57 WARN scheduler.TaskSetManager: Lost TID 12 (task 0.0:2)
14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 12]
14/09/04 12:53:57 ERROR scheduler.TaskSetManager: Task 0.0:2 failed 4 times; aborting job
14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 13]
14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
14/09/04 12:53:57 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38
Traceback (most recent call last):
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in &lt;module&gt;
    count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce
    vals = self.mapPartitions(func).collect()
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect
    bytesInJava = self._jrdd.collect().iterator()
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError14/09/04 12:53:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode
 [duplicate 14]
14/09/04 12:53:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException
org.apache.spark.TaskKilledException
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
: An error occurred while calling o24.collect.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:2 failed 4 times, most recent failure: Exception failure in TID 12 on host HDOP-M.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream
    for obj in iterator:
  File "/tmp/hadoop/yarn/local/usercache/root/filecache/11/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched
    for item in iterator:
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func
  File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f
SystemError: unknown opcode

        org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)
        org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)
        org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)
        org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        org.apache.spark.scheduler.Task.run(Task.scala:51)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
Driver stacktrace:
	at <a href="http://org.apache.spark.scheduler.DAGScheduler.org">org.apache.spark.scheduler.DAGScheduler.org</a>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

14/09/04 12:53:57 INFO cluster.YarnClientClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool </pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px">What other procedure can be done for fixing the problem.</pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"><br></pre><pre style="margin-top:0px;margin-bottom:0px;border:0px">Thanks</pre><pre style="margin-top:0px;margin-bottom:0px;border:0px">Oleg.</pre><pre style="margin-top:0px;margin-bottom:0px;border:0px"> </pre></div></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 4, 2014 at 5:36 AM, Andrew Or <span dir="ltr">&lt;<a href="mailto:andrew@databricks.com" target="_blank">andrew@databricks.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Oleg,<div><br></div><div>Your configuration looks alright to me. I haven't seen an "unknown opcode" System.error before in PySpark. This usually means you have corrupted .pyc files lying around (ones that belonged to an old python version, perhaps). What python version are you using? Are all your nodes running the same version of python? What happens if you just run bin/pyspark with the same command line arguments, and then do an "sc.parallelize(range(10)).count()", does it still fail?</div>

<div><br></div><div>Andrew</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-09-02 23:42 GMT-07:00 Oleg Ruchovets <span dir="ltr">&lt;<a href="mailto:oruchovets@gmail.com" target="_blank">oruchovets@gmail.com</a>&gt;</span>:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi I changed master to yarn but execution failed with exception again. I am using PySpark.<div><br></div>

<div><div>[root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn &nbsp;--num-executors 3 &nbsp;--driver-memory 4g --executor-memory 2g --executor-cores 1 &nbsp; examples/src/main/python/pi.py &nbsp; 1000</div>


<div>/usr/jdk64/jdk1.7.0_45/bin/java</div><div>::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf</div>


<div>-XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g</div><div>14/09/03 14:35:11 INFO spark.SecurityManager: Changing view acls to: root</div><div>14/09/03 14:35:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)</div>


<div>14/09/03 14:35:11 INFO slf4j.Slf4jLogger: Slf4jLogger started</div><div>14/09/03 14:35:11 INFO Remoting: Starting remoting</div><div>14/09/03 14:35:12 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:51707]</div>


<div>14/09/03 14:35:12 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:51707]</div><div>14/09/03 14:35:12 INFO spark.SparkEnv: Registering MapOutputTracker</div><div>14/09/03 14:35:12 INFO spark.SparkEnv: Registering BlockManagerMaster</div>


<div>14/09/03 14:35:12 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140903143512-5aab</div><div>14/09/03 14:35:12 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.</div><div>


14/09/03 14:35:12 INFO network.ConnectionManager: Bound socket to port 53216 with id = ConnectionManagerId(HDOP-B.AGT,53216)</div><div>14/09/03 14:35:12 INFO storage.BlockManagerMaster: Trying to register BlockManager</div>


<div>14/09/03 14:35:12 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:53216 with 2.3 GB RAM</div><div>14/09/03 14:35:12 INFO storage.BlockManagerMaster: Registered BlockManager</div><div>14/09/03 14:35:12 INFO spark.HttpServer: Starting HTTP Server</div>


<div>14/09/03 14:35:12 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 14:35:12 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:50624" target="_blank">SocketConnector@0.0.0.0:50624</a></div>

<div>
14/09/03 14:35:12 INFO broadcast.HttpBroadcast: Broadcast server started at <a href="http://10.193.1.76:50624" target="_blank">http://10.193.1.76:50624</a></div><div>14/09/03 14:35:12 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-fd7fdcb2-f45d-430f-95fa-afbc4f329b43</div>


<div>14/09/03 14:35:12 INFO spark.HttpServer: Starting HTTP Server</div><div>14/09/03 14:35:12 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 14:35:12 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:41773" target="_blank">SocketConnector@0.0.0.0:41773</a></div>


<div>14/09/03 14:35:13 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 14:35:13 INFO server.AbstractConnector: Started <a href="http://SelectChannelConnector@0.0.0.0:4040" target="_blank">SelectChannelConnector@0.0.0.0:4040</a></div>


<div>14/09/03 14:35:13 INFO ui.SparkUI: Started SparkUI at <a href="http://HDOP-B.AGT:4040" target="_blank">http://HDOP-B.AGT:4040</a></div><div>14/09/03 14:35:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>


<div>--args is deprecated. Use --arg instead.</div><div>14/09/03 14:35:14 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/<a href="http://10.193.1.72:8050" target="_blank">10.193.1.72:8050</a></div><div>

14/09/03 14:35:14 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6</div>
<div>14/09/03 14:35:14 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,</div><div>&nbsp; &nbsp; &nbsp; queueApplicationCount = 0, queueChildQueueCount = 0</div><div>14/09/03 14:35:14 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824</div>


<div>14/09/03 14:35:14 INFO yarn.Client: Preparing Local resources</div><div>14/09/03 14:35:14 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0036/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar</div>


<div>14/09/03 14:35:16 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0036/pi.py</div>


<div>14/09/03 14:35:16 INFO yarn.Client: Setting up the launch environment</div><div>14/09/03 14:35:16 INFO yarn.Client: Setting up container launch context</div><div>14/09/03 14:35:16 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-98b7d323-2faf-419a-a88d-1a0c549dc5d4\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -<a href="http://Dspark.app.name" target="_blank">Dspark.app.name</a>=\"PythonPi\", -Dspark.fileserver.uri=\"<a href="http://10.193.1.76:41773" target="_blank">http://10.193.1.76:41773</a>\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"51707\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"<a href="http://10.193.1.76:50624" target="_blank">http://10.193.1.76:50624</a>\", &nbsp;-Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null, &nbsp;--args &nbsp;'HDOP-B.AGT:51707' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)</div>


<div>14/09/03 14:35:16 INFO yarn.Client: Submitting application to ASM</div><div>14/09/03 14:35:16 INFO impl.YarnClientImpl: Submitted application application_1409559972905_0036</div><div>14/09/03 14:35:16 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div>


<div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div>


<div><br></div><div>14/09/03 14:35:17 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div>


<div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 14:35:18 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div>


<div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 14:35:19 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div>


<div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div>


<div><br></div><div>14/09/03 14:35:20 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div>


<div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 14:35:21 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div>


<div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 14:35:22 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div>


<div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: 0</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409726116517</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: RUNNING</div>


<div><br></div><div>14/09/03 14:35:24 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done</div><div>14/09/03 14:35:25 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:58976/user/Executor#-1831707618] with ID 1</div>


<div>14/09/03 14:35:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:44142 with 1178.1 MB RAM</div><div>14/09/03 14:35:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N1.AGT:45140/user/Executor#875812337] with ID 2</div>


<div>14/09/03 14:35:26 INFO storage.BlockManagerInfo: Registering block manager HDOP-N1.AGT:48513 with 1178.1 MB RAM</div><div>14/09/03 14:35:26 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N3.AGT:45380/user/Executor#1559437246] with ID 3</div>


<div>14/09/03 14:35:27 INFO storage.BlockManagerInfo: Registering block manager HDOP-N3.AGT:46616 with 1178.1 MB RAM</div><div>14/09/03 14:35:56 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38</div>


<div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)</div><div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)</div>


<div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Parents of final stage: List()</div><div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Missing parents: List()</div><div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents</div>


<div>14/09/03 14:35:56 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)</div><div>14/09/03 14:35:56 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks</div>


<div>14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:56 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 9 ms</div>


<div>14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:56 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 5 ms</div>


<div>14/09/03 14:35:56 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 5 ms</div>


<div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)</div><div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException</div><div>org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div><br></div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>


<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div>


<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 4 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div>


<div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms</div><div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)</div><div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException</div>


<div>org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div>


<div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div>


<div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div>


<div>&nbsp; &nbsp; for obj in iterator:</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div>


<div>&nbsp; &nbsp; for item in iterator:</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div><br></div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>


<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div>


<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 5 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div>


<div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 3 ms</div><div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 1]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 6 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:2)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 1]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 7 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 2]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 8 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:0)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 3]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 9 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:3)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 2]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 10 on executor 3: HDOP-N3.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:2)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 4]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 11 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:0)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 3]</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 12 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms</div>


<div>14/09/03 14:35:57 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:1)</div><div>14/09/03 14:35:57 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 5]</div><div>14/09/03 14:35:58 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 13 on executor 2: HDOP-N1.AGT (PROCESS_LOCAL)</div><div>14/09/03 14:35:58 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 3 ms</div>


<div>14/09/03 14:35:58 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:2)</div><div>14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 4]</div><div>14/09/03 14:35:58 ERROR scheduler.TaskSetManager: Task 0.0:2 failed 4 times; aborting job</div><div>14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0</div>


<div>14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled</div><div>14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 6]</div><div>14/09/03 14:35:58 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38</div>


<div>Traceback (most recent call last):</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in &lt;module&gt;</div><div>&nbsp; &nbsp; count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce</div><div>&nbsp; &nbsp; vals = self.mapPartitions(func).collect()</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect</div>


<div>&nbsp; &nbsp; bytesInJava = self._jrdd.collect().iterator()</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__</div><div>

&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value</div>
<div>py4j.protocol.Py4JJavaError14/09/03 14:35:58 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div>


<div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div>


<div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div>


<div>&nbsp; &nbsp; for obj in iterator:</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div>


<div>&nbsp; &nbsp; for item in iterator:</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 7]</div><div>: An error occurred while calling o24.collect.</div><div>: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:2 failed 4 times, most recent failure: Exception failure in TID 11 on host HDOP-N1.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>


<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/25/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>


<div>SystemError: unknown opcode</div><div><br></div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>


<div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div>


<div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>


<div>&nbsp; &nbsp; &nbsp; &nbsp; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; java.lang.Thread.run(Thread.java:744)</div>


<div>Driver stacktrace:</div><div><span style="white-space:pre-wrap">	</span>at <a href="http://org.apache.spark.scheduler.DAGScheduler.org" target="_blank">org.apache.spark.scheduler.DAGScheduler.org</a>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)</div>


<div><span style="white-space:pre-wrap">	</span>at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)</div><div><span style="white-space:pre-wrap">	</span>at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)</div><div><span style="white-space:pre-wrap">	</span>at scala.Option.foreach(Option.scala:236)</div>


<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)</div>


<div><span style="white-space:pre-wrap">	</span>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)</div><div><span style="white-space:pre-wrap">	</span>at akka.actor.ActorCell.invoke(ActorCell.scala:456)</div>
<div><span style="white-space:pre-wrap">	</span>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)</div><div><span style="white-space:pre-wrap">	</span>at akka.dispatch.Mailbox.run(Mailbox.scala:219)</div>
<div><span style="white-space:pre-wrap">	</span>at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)</div><div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)</div>


<div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)</div><div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)</div>


<div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)</div><div><br></div><div>14/09/03 14:35:58 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException</div>


<div>org.apache.spark.TaskKilledException</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div>


<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div><div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div>


<div>14/09/03 14:35:58 INFO cluster.YarnClientClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool&nbsp;</div><div><br></div><div>&nbsp; &nbsp;</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">


On Wed, Sep 3, 2014 at 1:53 PM, Oleg Ruchovets <span dir="ltr">&lt;<a href="mailto:oruchovets@gmail.com" target="_blank">oruchovets@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div dir="ltr">Hello Sandy , I changed to using yarn master but still got the exceptions:<div><br></div><div>What is the procedure to execute pyspark on yarn? is it required only to attached the command , or it is required to start spark processes also?</div>



<div><br></div><div><br></div><div><br></div><div><br><div><div>[root@HDOP-B spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master yarn://HDOP-N1.AGT:8032 --num-executors 3 &nbsp;--driver-memory 4g --executor-memory 2g --executor-cores 1 &nbsp; examples/src/main/python/pi.py &nbsp; 1000</div>


<div>
<div>/usr/jdk64/jdk1.7.0_45/bin/java</div><div>::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar:/etc/hadoop/conf</div>



<div>-XX:MaxPermSize=128m -Djava.library.path= -Xms4g -Xmx4g</div></div><div>14/09/03 13:48:48 INFO spark.SecurityManager: Changing view acls to: root</div><div>14/09/03 13:48:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)</div>



<div>14/09/03 13:48:49 INFO slf4j.Slf4jLogger: Slf4jLogger started</div><div>14/09/03 13:48:49 INFO Remoting: Starting remoting</div><div>14/09/03 13:48:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:34424]</div>



<div>14/09/03 13:48:49 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:34424]</div><div>14/09/03 13:48:49 INFO spark.SparkEnv: Registering MapOutputTracker</div><div>14/09/03 13:48:49 INFO spark.SparkEnv: Registering BlockManagerMaster</div>



<div>14/09/03 13:48:49 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140903134849-231c</div><div>14/09/03 13:48:49 INFO storage.MemoryStore: MemoryStore started with capacity 2.3 GB.</div><div>



14/09/03 13:48:49 INFO network.ConnectionManager: Bound socket to port 60647 with id = ConnectionManagerId(HDOP-B.AGT,60647)</div><div>14/09/03 13:48:49 INFO storage.BlockManagerMaster: Trying to register BlockManager</div>



<div>14/09/03 13:48:49 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:60647 with 2.3 GB RAM</div><div>14/09/03 13:48:49 INFO storage.BlockManagerMaster: Registered BlockManager</div><div>14/09/03 13:48:49 INFO spark.HttpServer: Starting HTTP Server</div>



<div>14/09/03 13:48:49 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 13:48:49 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:56549" target="_blank">SocketConnector@0.0.0.0:56549</a></div>


<div>
14/09/03 13:48:49 INFO broadcast.HttpBroadcast: Broadcast server started at <a href="http://10.193.1.76:56549" target="_blank">http://10.193.1.76:56549</a></div><div>14/09/03 13:48:49 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-90af1222-9ea8-4dd8-887a-343d09d44333</div>



<div>14/09/03 13:48:49 INFO spark.HttpServer: Starting HTTP Server</div><div>14/09/03 13:48:49 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 13:48:49 INFO server.AbstractConnector: Started <a href="http://SocketConnector@0.0.0.0:36512" target="_blank">SocketConnector@0.0.0.0:36512</a></div>



<div>14/09/03 13:48:50 INFO server.Server: jetty-8.y.z-SNAPSHOT</div><div>14/09/03 13:48:50 INFO server.AbstractConnector: Started <a href="http://SelectChannelConnector@0.0.0.0:4040" target="_blank">SelectChannelConnector@0.0.0.0:4040</a></div>



<div>14/09/03 13:48:50 INFO ui.SparkUI: Started SparkUI at <a href="http://HDOP-B.AGT:4040" target="_blank">http://HDOP-B.AGT:4040</a></div><div>14/09/03 13:48:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>


<div>
<div>--args is deprecated. Use --arg instead.</div></div><div>14/09/03 13:48:51 INFO client.RMProxy: Connecting to ResourceManager at HDOP-N1.AGT/<a href="http://10.193.1.72:8050" target="_blank">10.193.1.72:8050</a></div>


<div>14/09/03 13:48:51 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 6</div>
<div>14/09/03 13:48:51 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,</div><div><div>&nbsp; &nbsp; &nbsp; queueApplicationCount = 0, queueChildQueueCount = 0</div></div><div>
14/09/03 13:48:51 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 13824</div>
<div>14/09/03 13:48:51 INFO yarn.Client: Preparing Local resources</div><div>14/09/03 13:48:51 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0033/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar</div>



<div>14/09/03 13:48:53 INFO yarn.Client: Uploading file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to hdfs://HDOP-B.AGT:8020/user/root/.sparkStaging/application_1409559972905_0033/pi.py</div>



<div>14/09/03 13:48:53 INFO yarn.Client: Setting up the launch environment</div><div>14/09/03 13:48:53 INFO yarn.Client: Setting up container launch context</div><div>14/09/03 13:48:53 INFO yarn.Client: Command for starting the Spark ApplicationMaster: List($JAVA_HOME/bin/java, -server, -Xmx4096m, -Djava.io.tmpdir=$PWD/tmp, -Dspark.tachyonStore.folderName=\"spark-bdabb882-a2e0-46b6-8e87-90cc6e359d84\", -Dspark.executor.memory=\"2g\", -Dspark.executor.instances=\"3\", -Dspark.yarn.dist.files=\"file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py\", -Dspark.yarn.secondary.jars=\"\", -Dspark.submit.pyFiles=\"\", -Dspark.driver.host=\"HDOP-B.AGT\", -<a href="http://Dspark.app.name" target="_blank">Dspark.app.name</a>=\"PythonPi\", -Dspark.fileserver.uri=\"<a href="http://10.193.1.76:36512" target="_blank">http://10.193.1.76:36512</a>\", -Dspark.master=\"yarn-client\", -Dspark.driver.port=\"34424\", -Dspark.executor.cores=\"1\", -Dspark.httpBroadcast.uri=\"<a href="http://10.193.1.76:56549" target="_blank">http://10.193.1.76:56549</a>\", &nbsp;-Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null, &nbsp;--args &nbsp;'HDOP-B.AGT:34424' , --executor-memory, 2048, --executor-cores, 1, --num-executors , 3, 1&gt;, &lt;LOG_DIR&gt;/stdout, 2&gt;, &lt;LOG_DIR&gt;/stderr)</div>



<div>14/09/03 13:48:53 INFO yarn.Client: Submitting application to ASM</div><div>14/09/03 13:48:53 INFO impl.YarnClientImpl: Submitted application application_1409559972905_0033</div><div>14/09/03 13:48:53 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div>



<div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div>



<div><br></div><div>14/09/03 13:48:54 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div>



<div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 13:48:55 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div>



<div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 13:48:56 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div>



<div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div>



<div><br></div><div>14/09/03 13:48:57 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: -1</div><div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div>



<div><span style="white-space:pre-wrap">	</span> yarnAppState: ACCEPTED</div><div><br></div><div>14/09/03 13:48:58 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:&nbsp;</div><div><span style="white-space:pre-wrap">	</span> appMasterRpcPort: 0</div>



<div><span style="white-space:pre-wrap">	</span> appStartTime: 1409723333584</div><div><span style="white-space:pre-wrap">	</span> yarnAppState: RUNNING</div><div><br></div><div>14/09/03 13:49:00 INFO cluster.YarnClientClusterScheduler: YarnClientClusterScheduler.postStartHook done</div>



<div>14/09/03 13:49:01 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:57078/user/Executor#1595833626] with ID 1</div><div>14/09/03 13:49:02 INFO storage.BlockManagerInfo: Registering block manager HDOP-B.AGT:54579 with 1178.1 MB RAM</div>



<div>14/09/03 13:49:03 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N4.AGT:43121/user/Executor#-1266627304] with ID 2</div><div>14/09/03 13:49:03 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-N2.AGT:36952/user/Executor#1003961369] with ID 3</div>



<div>14/09/03 13:49:04 INFO storage.BlockManagerInfo: Registering block manager HDOP-N4.AGT:56891 with 1178.1 MB RAM</div><div>14/09/03 13:49:04 INFO storage.BlockManagerInfo: Registering block manager HDOP-N2.AGT:42381 with 1178.1 MB RAM</div>



<div>14/09/03 13:49:33 INFO spark.SparkContext: Starting job: reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38</div><div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Got job 0 (reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38) with 1000 output partitions (allowLocal=false)</div>



<div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38)</div><div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Parents of final stage: List()</div>



<div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Missing parents: List()</div><div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Submitting Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents</div>



<div>14/09/03 13:49:33 INFO scheduler.DAGScheduler: Submitting 1000 missing tasks from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)</div><div>14/09/03 13:49:33 INFO cluster.YarnClientClusterScheduler: Adding task set 0.0 with 1000 tasks</div>



<div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 0 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms</div>



<div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 1 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 5 ms</div>



<div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 2 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:33 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 5 ms</div>



<div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 3 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 5 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:2)</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException</div><div>org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div><br></div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>



<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div>



<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 4 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div>



<div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 4 ms</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException</div>


<div>
<div>org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div>



<div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div>



<div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div>



<div>&nbsp; &nbsp; for obj in iterator:</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div>



<div>&nbsp; &nbsp; for item in iterator:</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div><br></div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>



<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div>



<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 5 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)</div>



<div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException</div>



<div>org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div>



<div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div>



<div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div>



<div>&nbsp; &nbsp; for obj in iterator:</div><div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div>


<div>
<div>&nbsp; &nbsp; for item in iterator:</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div><br></div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>



<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div>



<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 6 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div>



<div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms</div><div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:3)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 1]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 7 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:2)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>
<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 1]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 8 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:1)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 1]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 9 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 6 (task 0.0:0)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 2]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 10 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 3 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:3)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>
<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 2]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 11 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 4 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 8 (task 0.0:2)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 2]</div></div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Starting task 0.0:2 as TID 12 on executor 1: HDOP-B.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Serialized task 0.0:2 as 501136 bytes in 3 ms</div>



<div>14/09/03 13:49:34 WARN scheduler.TaskSetManager: Lost TID 9 (task 0.0:1)</div><div>14/09/03 13:49:34 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/15/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 3]</div></div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID 13 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as 506276 bytes in 4 ms</div>



<div>14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 10 (task 0.0:0)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>
<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 3]</div></div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID 14 on executor 2: HDOP-N4.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as 369811 bytes in 4 ms</div>



<div>14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 11 (task 0.0:3)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 3]</div></div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Starting task 0.0:3 as TID 15 on executor 3: HDOP-N2.AGT (PROCESS_LOCAL)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Serialized task 0.0:3 as 506276 bytes in 3 ms</div>



<div>14/09/03 13:49:35 WARN scheduler.TaskSetManager: Lost TID 13 (task 0.0:1)</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>
<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div></div><div>&nbsp;[duplicate 4]</div><div>14/09/03 13:49:35 ERROR scheduler.TaskSetManager: Task 0.0:1 failed 4 times; aborting job</div><div>14/09/03 13:49:35 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0</div>



<div>14/09/03 13:49:35 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled</div><div>14/09/03 13:49:35 INFO scheduler.TaskSetManager: Loss was due to org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/19/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div>&nbsp;[duplicate 4]</div></div><div>14/09/03 13:49:35 INFO scheduler.DAGScheduler: Failed to run reduce at /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py:38</div>


<div>
<div>Traceback (most recent call last):</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in &lt;module&gt;</div><div>&nbsp; &nbsp; count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)</div>



</div><div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 619, in reduce</div><div>&nbsp; &nbsp; vals = self.mapPartitions(func).collect()</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 583, in collect</div>



<div>&nbsp; &nbsp; bytesInJava = self._jrdd.collect().iterator()</div></div><div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__</div>


<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value</div>
</div><div><div>py4j.protocol.Py4JJavaError: An error occurred while calling o24.collect.</div></div><div>: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:1 failed 4 times, most recent failure: Exception failure in TID 13 on host HDOP-N2.AGT: org.apache.spark.api.python.PythonException: Traceback (most recent call last):</div>


<div>
<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/worker.py", line 77, in main</div><div>&nbsp; &nbsp; serializer.dump_stream(func(split_index, iterator), outfile)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 191, in dump_stream</div><div>&nbsp; &nbsp; self.serializer.dump_stream(self._batched(iterator), stream)</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 123, in dump_stream</div><div>&nbsp; &nbsp; for obj in iterator:</div>



<div>&nbsp; File "/tmp/hadoop/yarn/local/usercache/root/filecache/23/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar/pyspark/serializers.py", line 180, in _batched</div><div>&nbsp; &nbsp; for item in iterator:</div>



<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/rdd.py", line 612, in func</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 36, in f</div>



<div>SystemError: unknown opcode</div><div><br></div></div><div><div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:115)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD$$anon$1.&lt;init&gt;(PythonRDD.scala:145)</div>



<div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:78)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.rdd.RDD.iterator(RDD.scala:229)</div>



<div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.scheduler.Task.run(Task.scala:51)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)</div>



<div>&nbsp; &nbsp; &nbsp; &nbsp; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div><div>&nbsp; &nbsp; &nbsp; &nbsp; java.lang.Thread.run(Thread.java:744)</div>



<div>Driver stacktrace:</div><div><span style="white-space:pre-wrap">	</span>at <a href="http://org.apache.spark.scheduler.DAGScheduler.org" target="_blank">org.apache.spark.scheduler.DAGScheduler.org</a>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1044)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1028)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1026)</div>



<div><span style="white-space:pre-wrap">	</span>at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)</div><div><span style="white-space:pre-wrap">	</span>at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1026)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:634)</div><div><span style="white-space:pre-wrap">	</span>at scala.Option.foreach(Option.scala:236)</div>



<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:634)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1229)</div>



<div><span style="white-space:pre-wrap">	</span>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)</div><div><span style="white-space:pre-wrap">	</span>at akka.actor.ActorCell.invoke(ActorCell.scala:456)</div>
<div><span style="white-space:pre-wrap">	</span>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)</div><div><span style="white-space:pre-wrap">	</span>at akka.dispatch.Mailbox.run(Mailbox.scala:219)</div>
<div><span style="white-space:pre-wrap">	</span>at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)</div><div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)</div>



<div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)</div><div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)</div>



<div><span style="white-space:pre-wrap">	</span>at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)</div><div><br></div></div></div><div>14/09/03 13:49:35 WARN scheduler.TaskSetManager: Loss was due to org.apache.spark.TaskKilledException</div>


<div>
<div>org.apache.spark.TaskKilledException</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:174)</div><div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)</div>



<div><span style="white-space:pre-wrap">	</span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)</div><div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div>



</div></div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 3, 2014 at 1:40 PM, Sandy Ryza <span dir="ltr">&lt;<a href="mailto:sandy.ryza@cloudera.com" target="_blank">sandy.ryza@cloudera.com</a>&gt;</span> wrote:<br>



<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Oleg. To run on YARN, simply set master to "yarn". &nbsp;The YARN configuration, located in a yarn-site.xml, determines where to look for the YARN ResourceManager.<div>



<br></div><div>PROCESS_LOCAL is orthogonal to the choice of cluster resource manager. A task is considered PROCESS_LOCAL when the executor it's running in happens to have the data it's processing cached.</div>
<div><br></div><div>If you're looking to get familiar with the kind of confusing web of terminology, this blog post might be helpful:&nbsp;</div><div><a href="http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/" target="_blank">http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/</a><span><font color="#888888"><br>




</font></span></div><span><font color="#888888"><div><br></div><div>-Sandy</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 2, 2014 at 9:51 PM, Oleg Ruchovets <span dir="ltr">&lt;<a href="mailto:oruchovets@gmail.com" target="_blank">oruchovets@gmail.com</a>&gt;</span> wrote:<br>




<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi ,&nbsp;<div>&nbsp; I change my command to :</div><div>&nbsp; ./bin/spark-submit --master spark://HDOP-B.AGT:7077 --num-executors 3 &nbsp;--driver-memory 4g --executor-memory 2g --executor-cores 1 &nbsp; examples/src/main/python/pi.py &nbsp; 1000</div>





<div>and it fixed the problem.</div><div><br></div><div>I still have couple of questions:&nbsp;</div><div><span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14.399999618530273px;line-height:20px">&nbsp; &nbsp;PROCESS_LOCAL is not Yarn execution , right? how should I configure the running on yarn? Should I exeture start-all script on all machine or only one? &nbsp;Where is the UI / LOGS of spark execution?</span></div>





<div><br></div><div><span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14.399999618530273px;line-height:20px"><br></span></div><div><br></div><div>
&nbsp; &nbsp;</div><br><table style="max-width:100%;border-spacing:0px;width:1483.199951171875px;margin-bottom:20px;border-width:1px 1px 1px 0px;border-top-style:solid;border-right-style:solid;border-bottom-style:solid;border-top-color:rgb(221,221,221);border-right-color:rgb(221,221,221);border-bottom-color:rgb(221,221,221);border-top-left-radius:4px;border-top-right-radius:4px;border-bottom-right-radius:4px;border-bottom-left-radius:4px;color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14.399999618530273px;line-height:20px">





<tbody><tr></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





152</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





152</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:14</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.2 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




0</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:09</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.9 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





39 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td></tr><tr>




<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:09</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.9 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





39 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




3</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">3</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:09</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.9 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





39 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">1 ms</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td></tr><tr>




<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





4</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





4</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:09</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.8 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





39 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




5</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">5</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:09</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.8 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





39 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">1 ms</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td></tr><tr>




<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





6</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





6</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:09</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.8 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





1 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




7</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">7</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:09</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.9 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





8</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





8</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:10</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.3 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




9</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">9</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:10</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.4 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)"></td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





10</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





10</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





PROCESS_LOCAL</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





HDOP-B.AGT</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





2014/09/03 12:35:10</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





0.3 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





1 ms</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221);background-color:rgb(249,249,249)">





</td></tr><tr><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">




11</td>
<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">11</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





SUCCESS</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">PROCESS_LOCAL</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">HDOP-B.AGT</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">2014/09/03 12:35:10</td>





<td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">0.3 s</td><td style="padding:4px 5px;vertical-align:top;border-top-width:1px;border-top-style:solid;border-top-color:rgb(221,221,221);border-left-width:1px;border-left-style:solid;border-left-color:rgb(221,221,221)">





</td></tr></tbody></table></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Sep 3, 2014 at 12:19 PM, Oleg Ruchovets <span dir="ltr">&lt;<a href="mailto:oruchovets@gmail.com" target="_blank">oruchovets@gmail.com</a>&gt;</span> wrote:<br>





<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Andrew.<div>&nbsp; &nbsp;what should I do to set master on yarn, can you please pointing me on command or documentation how to do it?</div>





<div><br></div><div><br></div><div>I am doing the following:</div>
<div>&nbsp; &nbsp;executed start-all.sh</div><div>&nbsp; &nbsp;[root@HDOP-B sbin]# ./start-all.sh&nbsp;</div><div>starting org.apache.spark.deploy.master.Master, logging to /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-HDOP-B.AGT.out</div>






<div>localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.</div><div>localhost: starting org.apache.spark.deploy.worker.Worker, logging to /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-HDOP-B.AGT.out</div>






<div><br></div><div><br></div><div>after execute the command:</div><div>&nbsp; &nbsp;&nbsp;./bin/spark-submit --master spark://HDOP-B.AGT:7077 examples/src/main/python/pi.py 1000</div><div><br></div><div><br></div>
<div>the result is the following:</div><div><br></div><div>&nbsp; &nbsp;/usr/jdk64/jdk1.7.0_45/bin/java</div><div>::/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/conf:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/lib/spark-assembly-1.0.1.2.1.3.0-563-hadoop2.4.0.2.1.3.0-563.jar</div>






<div>-XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m</div><div>14/09/03 12:10:06 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties</div><div>14/09/03 12:10:06 INFO SecurityManager: Changing view acls to: root</div>






<div>14/09/03 12:10:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)</div><div>14/09/03 12:10:07 INFO Slf4jLogger: Slf4jLogger started</div><div>





14/09/03 12:10:07 INFO Remoting: Starting remoting</div>
<div>14/09/03 12:10:07 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-B.AGT:38944]</div><div>14/09/03 12:10:07 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-B.AGT:38944]</div>






<div>14/09/03 12:10:07 INFO SparkEnv: Registering MapOutputTracker</div><div>14/09/03 12:10:07 INFO SparkEnv: Registering BlockManagerMaster</div><div>14/09/03 12:10:08 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140903121008-cf09</div>






<div>14/09/03 12:10:08 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.</div><div>14/09/03 12:10:08 INFO ConnectionManager: Bound socket to port 45041 with id = ConnectionManagerId(HDOP-B.AGT,45041)</div><div>






14/09/03 12:10:08 INFO BlockManagerMaster: Trying to register BlockManager</div><div>14/09/03 12:10:08 INFO BlockManagerInfo: Registering block manager HDOP-B.AGT:45041 with 294.9 MB RAM</div><div>14/09/03 12:10:08 INFO BlockManagerMaster: Registered BlockManager</div>






<div>14/09/03 12:10:08 INFO HttpServer: Starting HTTP Server</div><div>14/09/03 12:10:08 INFO HttpBroadcast: Broadcast server started at <a href="http://10.193.1.76:59336" target="_blank">http://10.193.1.76:59336</a></div>





<div>14/09/03 12:10:08 INFO HttpFileServer: HTTP File server directory is /tmp/spark-7bf5c3c3-1c02-41e8-9fb0-983e175dd45c</div>
<div>14/09/03 12:10:08 INFO HttpServer: Starting HTTP Server</div><div>14/09/03 12:10:08 INFO SparkUI: Started SparkUI at <a href="http://HDOP-B.AGT:4040" target="_blank">http://HDOP-B.AGT:4040</a></div><div>14/09/03 12:10:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>






<div>14/09/03 12:10:09 INFO Utils: Copying /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to /tmp/spark-4e252376-70cb-4171-bf2c-d804524e816c/pi.py</div><div>14/09/03 12:10:09 INFO SparkContext: Added file file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py at <a href="http://10.193.1.76:45893/files/pi.py" target="_blank">http://10.193.1.76:45893/files/pi.py</a> with timestamp 1409717409277</div>






<div>14/09/03 12:10:09 INFO AppClient$ClientActor: Connecting to master spark://HDOP-B.AGT:7077...</div><div>14/09/03 12:10:09 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140903121009-0000</div>






<div>14/09/03 12:10:09 INFO AppClient$ClientActor: Executor added: app-20140903121009-0000/0 on worker-20140903120712-HDOP-B.AGT-51161 (HDOP-B.AGT:51161) with 8 cores</div><div>14/09/03 12:10:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140903121009-0000/0 on hostPort HDOP-B.AGT:51161 with 8 cores, 512.0 MB RAM</div>






<div>14/09/03 12:10:09 INFO AppClient$ClientActor: Executor updated: app-20140903121009-0000/0 is now RUNNING</div><div>14/09/03 12:10:12 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@HDOP-B.AGT:38143/user/Executor#1295757828] with ID 0</div>






<div>14/09/03 12:10:12 INFO BlockManagerInfo: Registering block manager HDOP-B.AGT:38670 with 294.9 MB RAM</div><div><div>Traceback (most recent call last):</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in &lt;module&gt;</div>






<div>&nbsp; &nbsp; count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 271, in parallelize</div><div>&nbsp; &nbsp; jrdd = readRDDFromFile(self._jsc, tempFile.name, numSlices)</div>






<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value</div>






<div>py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.</div></div><div>: java.lang.OutOfMemoryError: Java heap space</div><div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$.readRDDFromFile(PythonRDD.scala:279)</div>






<div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.readRDDFromFile(PythonRDD.scala)</div><div><span style="white-space:pre-wrap">	</span>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div>






<div><span style="white-space:pre-wrap">	</span>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)</div><div><span style="white-space:pre-wrap">	</span>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</div>






<div><span style="white-space:pre-wrap">	</span>at java.lang.reflect.Method.invoke(Method.java:606)</div><div><span style="white-space:pre-wrap">	</span>at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)</div>






<div><span style="white-space:pre-wrap">	</span>at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)</div><div><span style="white-space:pre-wrap">	</span>at py4j.Gateway.invoke(Gateway.java:259)</div>
<div><span style="white-space:pre-wrap">	</span>at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)</div><div><span style="white-space:pre-wrap">	</span>at py4j.commands.CallCommand.execute(CallCommand.java:79)</div>






<div><span style="white-space:pre-wrap">	</span>at py4j.GatewayConnection.run(GatewayConnection.java:207)</div><div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div><div><br>
</div><div><br></div><div><br></div></div><div>What should I do to fix the issue&nbsp;</div><div><br></div><div>Thanks</div><span><font color="#888888"><div>Oleg.</div></font></span></div><div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 2, 2014 at 10:32 PM, Andrew Or <span dir="ltr">&lt;<a href="mailto:andrew@databricks.com" target="_blank">andrew@databricks.com</a>&gt;</span> wrote:<br>






<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Oleg,<div><br></div><div>If you are running Spark on a yarn cluster, you should set --master to yarn. By default this runs in client mode, which redirects all output of your application to your console. This is failing because it is trying to connect to a standalone master that you probably did not start. I am somewhat puzzled as to how you ran into an OOM from this configuration, however. Does this problem still occur if you set the correct master?</div>







<div><br></div><div>-Andrew</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-09-02 2:42 GMT-07:00 Oleg Ruchovets <span dir="ltr">&lt;<a href="mailto:oruchovets@gmail.com" target="_blank">oruchovets@gmail.com</a>&gt;</span>:<div>






<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi ,&nbsp;<div>&nbsp; &nbsp;I've installed pyspark on hpd hortonworks cluster.&nbsp;</div><div>&nbsp; Executing pi example:</div>







<div><br></div><div>command:</div><div>&nbsp; &nbsp; &nbsp; &nbsp;spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563]# ./bin/spark-submit --master spark://<a href="http://10.193.1.71:7077" target="_blank">10.193.1.71:7077</a> &nbsp; examples/src/main/python/pi.py &nbsp; 1000<br>








</div><div><br></div><div>exception:</div><div><br></div><div>&nbsp; &nbsp; 14/09/02 17:34:02 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties</div><div>
14/09/02 17:34:02 INFO SecurityManager: Changing view acls to: root</div><div>14/09/02 17:34:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)</div>








<div>14/09/02 17:34:02 INFO Slf4jLogger: Slf4jLogger started</div><div>14/09/02 17:34:02 INFO Remoting: Starting remoting</div><div>14/09/02 17:34:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@HDOP-M.AGT:41059]</div>








<div>14/09/02 17:34:03 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@HDOP-M.AGT:41059]</div><div>14/09/02 17:34:03 INFO SparkEnv: Registering MapOutputTracker</div><div>14/09/02 17:34:03 INFO SparkEnv: Registering BlockManagerMaster</div>








<div>14/09/02 17:34:03 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140902173403-cda8</div><div>14/09/02 17:34:03 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.</div><div>14/09/02 17:34:03 INFO ConnectionManager: Bound socket to port 34931 with id = ConnectionManagerId(HDOP-M.AGT,34931)</div>








<div>14/09/02 17:34:03 INFO BlockManagerMaster: Trying to register BlockManager</div><div>14/09/02 17:34:03 INFO BlockManagerInfo: Registering block manager HDOP-M.AGT:34931 with 294.9 MB RAM</div><div>14/09/02 17:34:03 INFO BlockManagerMaster: Registered BlockManager</div>








<div>14/09/02 17:34:03 INFO HttpServer: Starting HTTP Server</div><div>14/09/02 17:34:03 INFO HttpBroadcast: Broadcast server started at <a href="http://10.193.1.71:54341" target="_blank">http://10.193.1.71:54341</a></div>







<div>14/09/02 17:34:03 INFO HttpFileServer: HTTP File server directory is /tmp/spark-77c7a7dc-181e-4069-a014-8103a6a6330a</div>
<div>14/09/02 17:34:03 INFO HttpServer: Starting HTTP Server</div><div>14/09/02 17:34:04 INFO SparkUI: Started SparkUI at <a href="http://HDOP-M.AGT:4040" target="_blank">http://HDOP-M.AGT:4040</a></div><div>14/09/02 17:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>








<div>14/09/02 17:34:04 INFO Utils: Copying /root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py to /tmp/spark-f2e0cc0f-59cb-4f6c-9d48-f16205a40c7e/pi.py</div><div>14/09/02 17:34:04 INFO SparkContext: Added file file:/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py at <a href="http://10.193.1.71:52938/files/pi.py" target="_blank">http://10.193.1.71:52938/files/pi.py</a> with timestamp 1409650444941</div>








<div>14/09/02 17:34:05 INFO AppClient$ClientActor: Connecting to master spark://10.193.1.71:7077...</div><div>14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:05 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:25 INFO AppClient$ClientActor: Connecting to master spark://10.193.1.71:7077...</div><div>14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>14/09/02 17:34:25 WARN AppClient$ClientActor: Could not connect to akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://<a href="http://sparkMaster@10.193.1.71:7077" target="_blank">sparkMaster@10.193.1.71:7077</a>]</div>








<div>Traceback (most recent call last):</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/examples/src/main/python/pi.py", line 38, in &lt;module&gt;</div><div>&nbsp; &nbsp; count = sc.parallelize(xrange(1, n+1), slices).map(f).reduce(add)</div>








<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/pyspark/context.py", line 271, in parallelize</div><div>&nbsp; &nbsp; jrdd = readRDDFromFile(self._jsc, tempFile.name, numSlices)</div><div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__</div>








<div>&nbsp; File "/root/spark-1.0.1.2.1.3.0-563-bin-2.4.0.2.1.3.0-563/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value</div><div>py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.readRDDFromFile.</div>








<div>: java.lang.OutOfMemoryError: GC overhead limit exceeded</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD$.readRDDFromFile(PythonRDD.scala:279)</div><div><span style="white-space:pre-wrap">	</span>at org.apache.spark.api.python.PythonRDD.readRDDFromFile(PythonRDD.scala)</div>








<div><span style="white-space:pre-wrap">	</span>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div><div><span style="white-space:pre-wrap">	</span>at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)</div>








<div><span style="white-space:pre-wrap">	</span>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</div><div><span style="white-space:pre-wrap">	</span>at java.lang.reflect.Method.invoke(Method.java:606)</div>








<div><span style="white-space:pre-wrap">	</span>at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)</div><div><span style="white-space:pre-wrap">	</span>at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)</div>








<div><span style="white-space:pre-wrap">	</span>at py4j.Gateway.invoke(Gateway.java:259)</div><div><span style="white-space:pre-wrap">	</span>at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)</div>
<div><span style="white-space:pre-wrap">	</span>at py4j.commands.CallCommand.execute(CallCommand.java:79)</div><div><span style="white-space:pre-wrap">	</span>at py4j.GatewayConnection.run(GatewayConnection.java:207)</div>








<div><span style="white-space:pre-wrap">	</span>at java.lang.Thread.run(Thread.java:744)</div><div><br></div><div><br></div><div><br></div><div>Question:&nbsp;</div><div>&nbsp; &nbsp; how can I know spark master and port? Where is it defined?</div>








<div><br></div><div>Thanks</div><span><font color="#888888"><div>Oleg.</div></font></span></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div>
</blockquote></div><br></div>
</div></blockquote></body></html>
Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message