spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurav Sinha <sauravsinh...@gmail.com>
Subject Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Date Thu, 06 Oct 2016 09:51:03 GMT
Hi User,

I am trying to run spark streaming job in yarn-cluster mode.

It is failing with code

val conf = new SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")

val ssc = new StreamingContext(conf,
Seconds(properties.getProperty("batchDurationInSeconds").toInt));


org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
running on a cluster. Deployment to YARN is not supported directly by
SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" org.apache.spark.SparkException: Detected
yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
not supported directly by SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Please help.
-- 
Thanks and Regards,

Saurav Sinha

Contact: 9742879062

Mime
View raw message