spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeetendra Gangele <gangele...@gmail.com>
Subject Spark on YARN
Date Thu, 30 Jul 2015 06:51:01 GMT
I am running below command this is default spark PI program but this is not
running all the log are going in stderr but at the terminal job is
succeeding .I guess there are con issue job it not at all launching

/bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10


Complete log

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal
handlers for [TERM, HUP, INT]
15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId:
appattempt_1438090734187_0010_000001
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user
application in a separate Thread
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark
context initialization
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark
context initialization ...
15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
15/07/30 12:13:33 WARN spark.SparkConf:
SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with conf/spark-defaults.conf to set defaults for an
application
 - ./spark-submit with --driver-java-options to set -X options for a driver
 - spark.executor.extraJavaOptions to set -X options for executors
 - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons
(master or worker)

15/07/30 12:13:33 WARN spark.SparkConf: Setting
'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/07/30 12:13:33 WARN spark.SparkConf: Setting
'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/07/30 12:13:33 INFO Remoting: Starting remoting
15/07/30 12:13:34 INFO Remoting: Remoting started; listening on
addresses :[akka.tcp://sparkDriver@10.21.1.77:53411]
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'sparkDriver' on port 53411.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local
directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with
capacity 246.0 MB
15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server
directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:52507
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP
file server' on port 52507.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:59596
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'SparkUI' on port 59596.
15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port
43354.
15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering
block manager 10.21.1.77:43354 with 246.0 MB RAM,
BlockManagerId(driver, 10.21.1.77, 43354)
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
15/07/30 12:13:34 INFO
cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster
registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager
at hadoop-1/10.21.1.77:8030
15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor
containers, each with 1 cores and 1408 MB memory including 384 MB
overhead
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress
reporter thread - sleep time : 5000
15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status:
SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final
status was reported.)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering
ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called
before final status was reported.)
15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to
be successfully unregistered.
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging
directory .sparkStaging/application_1438090734187_0010
15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Deleting directory
/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
15/07/30 12:13:35 INFO util.Utils: Deleting directory
/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c

Mime
View raw message