spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: [EXTERNAL] Urgent Help - Py Spark submit error
Date Sat, 15 May 2021 18:03:48 GMT
This is an interesting one.

I have never tried to add --files ...

spark-submit --master yarn --deploy-mode client --files
/etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

Rather, under $SPARK_HOME/conf, I create soft links to the needed XML files
as below

/d4T/hduser/spark-3.1.1-bin-hadoop3.2/conf> ls -lhaF | grep ^l
lrwxrwxrwx  1 hduser hadoop   50 Mar  3 08:08 core-site.xml ->
/home/hduser/hadoop-3.1.0/etc/hadoop/core-site.xml
lrwxrwxrwx  1 hduser hadoop   45 Mar  3 08:07 hbase-site.xml ->
/data6/hduser/hbase-1.2.6/conf/hbase-site.xml
lrwxrwxrwx  1 hduser hadoop   50 Mar  3 08:08 hdfs-site.xml ->
/home/hduser/hadoop-3.1.0/etc/hadoop/hdfs-site.xml
lrwxrwxrwx  1 hduser hadoop   43 Mar  3 08:07 hive-site.xml ->
/data6/hduser/hive-3.0.0/conf/hive-site.xml

This works

HTH



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sat, 15 May 2021 at 18:32, KhajaAsmath Mohammed <mdkhajaasmath@gmail.com>
wrote:

> Thanks everyone. I was able to resolve this.
>
> Here is what I did. Just passed conf file using —files option.
>
> Mistake that I did was reading the json conf file before creating spark
> session . Reading if after creating spark session helped it. Thanks once
> again for your valuable suggestions
>
> Thanks,
> Asmath
>
> On May 15, 2021, at 8:12 AM, Sean Owen <srowen@gmail.com> wrote:
>
> 
> If code running on the executors need some local file like a config file,
> then it does have to be passed this way. That much is normal.
>
> On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta <gourav.sengupta@gmail.com>
> wrote:
>
>> Hi,
>>
>> once again lets start with the requirement. Why are you trying to pass
>> xml and json files to SPARK instead of reading them in SPARK?
>> Generally when people pass on files they are python or jar files.
>>
>> Regards,
>> Gourav
>>
>

Mime
View raw message