spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rajat kumar <kumar.rajat20...@gmail.com>
Subject Running pyspark job from virtual environment
Date Sat, 16 Jan 2021 18:16:23 GMT
Hey Users,

I want to run spark job from virtual environment using Python.

Please note I am creating virtual env (using python3 -m venv env)

I see that there are 3 variables for PYTHON which we have to set:
PYTHONPATH
PYSPARK_DRIVER_PYTHON
PYSPARK_PYTHON

I have 2 doubts:
1. If i want to use Virtual env, do I need to point python path of virtual
environment to all these variables?
2. Should I set these variables in spark-env.sh or should I set them using
export statements.

Regards
Rajat

Mime
View raw message