spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: PyCharm, Running spark-submit calling jars and a package at run time
Date Fri, 08 Jan 2021 16:27:35 GMT
I don't see anywhere that you provide 'sparkstuff'? how would the Spark app
have this code otherwise?

On Fri, Jan 8, 2021 at 10:20 AM Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> Thanks Riccardo.
>
> I am well aware of the submission form
>
> However, my question relates to doing submission within PyCharm itself.
>
> This is what I do at Pycharm *terminal* to invoke the module python
>
> spark-submit --jars
> ..\lib\spark-bigquery-with-dependencies_2.12-0.18.0.jar \
>  --packages com.github.samelamin:spark-bigquery_2.11:0.2.6
> analyze_house_prices_GCP.py
>
> However, at terminal run it does not pickup import dependencies in the
> code!
>
> Traceback (most recent call last):
>   File
> "C:/Users/admin/PycharmProjects/pythonProject2/DS/src/analyze_house_prices_GCP.py",
> line 8, in <module>
>     import sparkstuff as s
> ModuleNotFoundError: No module named 'sparkstuff'
>
> The python code is attached, pretty simple
>
> Thanks
>
>
>
>

Mime
View raw message