This is an interesting one.

I have never tried to add --files ...

spark-submit --master yarn --deploy-mode client --files /etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

Rather, under $SPARK_HOME/conf, I create soft links to the needed XML files as below

/d4T/hduser/spark-3.1.1-bin-hadoop3.2/conf> ls -lhaF | grep ^l  
lrwxrwxrwx  1 hduser hadoop   50 Mar  3 08:08 core-site.xml -> /home/hduser/hadoop-3.1.0/etc/hadoop/core-site.xml
lrwxrwxrwx  1 hduser hadoop   45 Mar  3 08:07 hbase-site.xml -> /data6/hduser/hbase-1.2.6/conf/hbase-site.xml
lrwxrwxrwx  1 hduser hadoop   50 Mar  3 08:08 hdfs-site.xml -> /home/hduser/hadoop-3.1.0/etc/hadoop/hdfs-site.xml
lrwxrwxrwx  1 hduser hadoop   43 Mar  3 08:07 hive-site.xml -> /data6/hduser/hive-3.0.0/conf/hive-site.xml

This works 

HTH



   view my Linkedin profile

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Sat, 15 May 2021 at 18:32, KhajaAsmath Mohammed <mdkhajaasmath@gmail.com> wrote:
Thanks everyone. I was able to resolve this. 

Here is what I did. Just passed conf file using —files option.

Mistake that I did was reading the json conf file before creating spark session . Reading if after creating spark session helped it. Thanks once again for your valuable suggestions 

Thanks,
Asmath

On May 15, 2021, at 8:12 AM, Sean Owen <srowen@gmail.com> wrote:


If code running on the executors need some local file like a config file, then it does have to be passed this way. That much is normal.

On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta <gourav.sengupta@gmail.com> wrote:
Hi,

once again lets start with the requirement. Why are you trying to pass xml and json files to SPARK instead of reading them in SPARK? 
Generally when people pass on files they are python or jar files.

Regards,
Gourav