spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 朱旻 <>
Subject Re:Re: Re: spark uploading resource error
Date Tue, 10 May 2016 09:09:22 GMT
i solved the problem.
spark-submit changed the HADOOP_CONF_DIR to spark/conf and was corrent
but using java *****...  didn't change the HADOOP_CONF_DIR. it still use hadoop/etc/hadoop.

At 2016-05-10 16:39:47, "Saisai Shao" <> wrote:

The code is in Client.scala under yarn sub-module (see the below link). Maybe you need to
check the vendor version about their changes to the Apache Spark code.


On Tue, May 10, 2016 at 4:17 PM, 朱旻 <> wrote:

it was a product sold by huawei . name is FusionInsight. it says spark was 1.3 with hadoop

where can i find the code or config file which define the files to be uploaded?

At 2016-05-10 16:06:05, "Saisai Shao" <> wrote:

What is the version of Spark are you using? From my understanding, there's no code in yarn#client
will upload "__hadoop_conf__" into distributed cache.

On Tue, May 10, 2016 at 3:51 PM, 朱旻 <> wrote:

hi all:
I found a problem using spark .
WHEN I use spark-submit to launch a task. it works

spark-submit --num-executors 8 --executor-memory 8G --class
 --master yarn-cluster /home/nssbatch/nss_schedual/jar/SparkBigtableJoinSqlJava.jar /user/nss/nss-20151018-pfsjnl-004_024.txt
/user/nss/output_join /user/nss/output_join2

but when i use the command created by spark-class  as below

-Dzookeeper.kinit=/home/nssbatch/huaweiclient/hadoopclient/KrbClient/kerberos/bin/kinit -cp
org.apache.spark.deploy.SparkSubmit --master yarn-cluster --class
--num-executors 8 --executor-memory 8G /home/nssbatch/nss_schedual/jar/SparkBigtableJoinSqlJava.jar
/user/nss/nss-20151018-pfsjnl-004_024.txt /user/nss/output_join /user/nss/output_join2

it didn't work. 
i compare the log.and found that: 

16/05/10 22:23:24 INFO Client: Uploading resource file:/tmp/spark-a4457754-7183-44ce-bd0d-32a071757c92/
-> hdfs://hacluster/user/nss/.sparkStaging/application_1462442311990_0057/

the conf_file uploaded into hdfs was different.

why is this happened?
where can i find the resource file to be uploading?



View raw message