spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aditya" <aditya.calangut...@augmentiq.co.in>
Subject Re: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Date Fri, 07 Oct 2016 09:20:33 GMT
Hi Saurav,

Please share spark-submit command which you used.

On Friday 07 October 2016 02:41 PM, Saurav Sinha wrote:
> I am submitting job by spark-submit but still it is giving message.
>
> Please use spark-submit.
>
> Can any one give me resone for this error.
>
> Thanks,
> Saurav Sinha
>
> On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sauravsinha76@gmail.com 
> <mailto:sauravsinha76@gmail.com>> wrote:
>
>     I did not get you I am submitting job by spark-submit but still it
>     is giving message.
>
>     Please use spark-submit.
>
>     On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen <sowen@cloudera.com
>     <mailto:sowen@cloudera.com>> wrote:
>
>         Did you search on the internet / read the docs? the error here
>         tells you the problem exactly. Your other question is easily
>         answerable by looking up PermGen.
>
>         On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha
>         <sauravsinha76@gmail.com <mailto:sauravsinha76@gmail.com>> wrote:
>
>             Hi User,
>
>             I am trying to run spark streaming job in yarn-cluster mode.
>
>             It is failing with code
>
>             val conf = new
>             SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")
>
>             val ssc = new StreamingContext(conf,
>             Seconds(properties.getProperty("batchDurationInSeconds").toInt));
>
>
>             org.apache.spark.SparkException: Detected yarn-cluster
>             mode, but isn't running on a cluster. Deployment to YARN
>             is not supported directly by SparkContext. Please use
>             spark-submit.
>             at
>             org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>             at
>             org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
>             at
>             org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
>             at
>             com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>             at
>             com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>             at
>             com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>             at scala.Option.getOrElse(Option.scala:120)
>             at
>             org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
>             at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>             at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>             at
>             sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>             at
>             sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>             at java.lang.reflect.Method.invoke(Method.java:498)
>             at
>             org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
>             at
>             org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>             at
>             org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>             at
>             org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>             at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>             Exception in thread "main"
>             org.apache.spark.SparkException: Detected yarn-cluster
>             mode, but isn't running on a cluster. Deployment to YARN
>             is not supported directly by SparkContext. Please use
>             spark-submit.
>             at
>             org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>             at
>             org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
>             at
>             org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
>             at
>             com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>             at
>             com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>             at
>             com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>             at scala.Option.getOrElse(Option.scala:120)
>             at
>             org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
>             at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>             at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>             at
>             sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>             at
>             sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>             at java.lang.reflect.Method.invoke(Method.java:498)
>             at
>             org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
>             at
>             org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
>             at
>             org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>             at
>             org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>             at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>             Please help.
>             -- 
>             Thanks and Regards,
>
>             Saurav Sinha
>
>             Contact: 9742879062
>
>
>
>
>     -- 
>     Thanks and Regards,
>
>     Saurav Sinha
>
>     Contact: 9742879062
>
>
>
>
> -- 
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062




Mime
View raw message