spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Riccardo Ferrari <ferra...@gmail.com>
Subject Re: PyCharm, Running spark-submit calling jars and a package at run time
Date Fri, 08 Jan 2021 16:38:23 GMT
I think spark checks the python path env variable. Need to provide that.
Of course that works in local mode only

On Fri, Jan 8, 2021, 5:28 PM Sean Owen <srowen@gmail.com> wrote:

> I don't see anywhere that you provide 'sparkstuff'? how would the Spark
> app have this code otherwise?
>
> On Fri, Jan 8, 2021 at 10:20 AM Mich Talebzadeh <mich.talebzadeh@gmail.com>
> wrote:
>
>> Thanks Riccardo.
>>
>> I am well aware of the submission form
>>
>> However, my question relates to doing submission within PyCharm itself.
>>
>> This is what I do at Pycharm *terminal* to invoke the module python
>>
>> spark-submit --jars
>> ..\lib\spark-bigquery-with-dependencies_2.12-0.18.0.jar \
>>  --packages com.github.samelamin:spark-bigquery_2.11:0.2.6
>> analyze_house_prices_GCP.py
>>
>> However, at terminal run it does not pickup import dependencies in the
>> code!
>>
>> Traceback (most recent call last):
>>   File
>> "C:/Users/admin/PycharmProjects/pythonProject2/DS/src/analyze_house_prices_GCP.py",
>> line 8, in <module>
>>     import sparkstuff as s
>> ModuleNotFoundError: No module named 'sparkstuff'
>>
>> The python code is attached, pretty simple
>>
>> Thanks
>>
>>
>>
>>

Mime
View raw message