spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
Subject Re: [EXTERNAL] Urgent Help - Py Spark submit error
Date Sat, 15 May 2021 17:31:44 GMT
Thanks everyone. I was able to resolve this. 

Here is what I did. Just passed conf file using —files option.

Mistake that I did was reading the json conf file before creating spark session . Reading
if after creating spark session helped it. Thanks once again for your valuable suggestions


Thanks,
Asmath

> On May 15, 2021, at 8:12 AM, Sean Owen <srowen@gmail.com> wrote:
> 
> 
> If code running on the executors need some local file like a config file, then it does
have to be passed this way. That much is normal.
> 
>> On Sat, May 15, 2021 at 1:41 AM Gourav Sengupta <gourav.sengupta@gmail.com>
wrote:
>> Hi,
>> 
>> once again lets start with the requirement. Why are you trying to pass xml and json
files to SPARK instead of reading them in SPARK? 
>> Generally when people pass on files they are python or jar files.
>> 
>> Regards,
>> Gourav

Mime
View raw message