spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Nemuri <nhsande...@gmail.com>
Subject Re: SparkSession via HS2 - Error -spark.yarn.jars not read
Date Wed, 05 Jul 2017 10:34:56 GMT
STS will refer spark-thrift-sparkconf.conf, Can you check if the
spark.yarn.jars exists in this file?



On Wed, Jul 5, 2017 at 2:01 PM, Sudha KS <Sudha.KS@fuzzylogix.com> wrote:

> The property “spark.yarn.jars” available via /usr/hdp/current/spark2-
> client/conf/spark-default.conf
>
>
>
> spark.yarn.jars hdfs://ambari03.fuzzyl.com:8020/hdp/apps/2.6.1.0-129/
> spark2
>
>
>
>
>
> Is there any other way to set/read/pass this property “spark.yarn.jars” ?
>
>
>
> *From:* Sudha KS [mailto:Sudha.KS@fuzzylogix.com]
> *Sent:* Wednesday, July 5, 2017 1:51 PM
> *To:* user@spark.apache.org
> *Subject:* SparkSession via HS2 - Error -spark.yarn.jars not read
>
>
>
> Why does “spark.yarn.jars” property not read, in this HDP 2.6 , Spark2.1.1
> cluster:
>
> 0: jdbc:hive2://localhost:10000/db> set spark.yarn.jars;
>
> +-----------------------------------------------------------
> -------------------+--+
>
> |                                     set
> |
>
> +-----------------------------------------------------------
> -------------------+--+
>
> | spark.yarn.jars=hdfs://ambari03.fuzzyl.com:8020/hdp/apps/2.
> 6.1.0-129/spark2  |
>
> +-----------------------------------------------------------
> -------------------+--+
>
> 1 row selected (0.101 seconds)
>
> 0: jdbc:hive2://localhost:10000/db>
>
>
>
>
>
>
>
> Error during launch of a SparkSession via HS2:
>
> Caused by: java.lang.IllegalStateException: Library directory
> '/hadoop/yarn/local/usercache/hive/appcache/application_
> 1499235958765_0042/container_e04_1499235958765_0042_01_
> 000005/assembly/target/scala-2.11/jars' does not exist; make sure Spark
> is built.
>
>         at org.apache.spark.launcher.CommandBuilderUtils.checkState(
> CommandBuilderUtils.java:260)
>
>         at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(
> CommandBuilderUtils.java:380)
>
>         at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(
> YarnCommandBuilderUtils.scala:38)
>
>         at org.apache.spark.deploy.yarn.Client.prepareLocalResources(
> Client.scala:570)
>
>         at org.apache.spark.deploy.yarn.Client.
> createContainerLaunchContext(Client.scala:895)
>
>         at org.apache.spark.deploy.yarn.Client.submitApplication(
> Client.scala:171)
>
>         at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.
> start(YarnClientSchedulerBackend.scala:56)
>
>         at org.apache.spark.scheduler.TaskSchedulerImpl.start(
> TaskSchedulerImpl.scala:156)
>
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
>
>         at org.apache.spark.SparkContext$.getOrCreate(SparkContext.
> scala:2320)
>
>         at org.apache.spark.sql.SparkSession$Builder$$anonfun$
> 6.apply(SparkSession.scala:868)
>
>         at org.apache.spark.sql.SparkSession$Builder$$anonfun$
> 6.apply(SparkSession.scala:860)
>
>         at scala.Option.getOrElse(Option.scala:121)
>
>         at org.apache.spark.sql.SparkSession$Builder.
> getOrCreate(SparkSession.scala:860)
>
>         at SparkHiveUDTF.sparkJob(SparkHiveUDTF.java:97)
>
>         at SparkHiveUDTF.process(SparkHiveUDTF.java:78)
>
>         at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(
> UDTFOperator.java:109)
>
>         at org.apache.hadoop.hive.ql.exec.Operator.forward(
> Operator.java:841)
>
>         at org.apache.hadoop.hive.ql.exec.SelectOperator.process(
> SelectOperator.java:88)
>
>         at org.apache.hadoop.hive.ql.exec.Operator.forward(
> Operator.java:841)
>
>         at org.apache.hadoop.hive.ql.exec.TableScanOperator.
> process(TableScanOperator.java:133)
>
>         at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.
> forward(MapOperator.java:170)
>
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(
> MapOperator.java:555)
>
>         ... 18 more
>
>
>
>
>
>
>
>
>



-- 
*  Regards*
*  Sandeep Nemuri*

Mime
View raw message