spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From karan alang <karan.al...@gmail.com>
Subject spark-submit not running on macbook pro
Date Tue, 17 Aug 2021 05:56:14 GMT
Hello Experts,

i'm trying to run spark-submit on my macbook pro(commandline or using
PyCharm), and it seems to be giving error ->

Exception: Java gateway process exited before sending its port number

i've tried setting values to variable in the program (based on the
recommendations by people on the internet), but the problem still remains.

Any pointers on how to resolve this issue?

# explicitly setting environment variables
os.environ["JAVA_HOME"] =
"/Library/Java/JavaVirtualMachines/applejdk-11.0.7.10.1.jdk/Contents/Home"
os.environ["PYTHONPATH"] =
"/usr/local/Cellar/apache-spark/3.1.2/libexec//python/lib/py4j-0.10.4-src.zip:/usr/local/Cellar/apache-spark/3.1.2/libexec//python/:"
os.environ["PYSPARK_SUBMIT_ARGS"]="--master local[2] pyspark-shell"

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm
CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py",
line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute
the script
  File "/Applications/PyCharm
CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py",
line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File
"/Users/karanalang/Documents/Technology/StructuredStreamin_Udemy/Spark-Streaming-In-Python-master/00-HelloSparkSQL/HelloSparkSQL.py",
line 12, in <module>
    spark = SparkSession.builder.master("local[*]").getOrCreate()
  File
"/Users/karanalang/.conda/envs/PythonLeetcode/lib/python3.9/site-packages/pyspark/sql/session.py",
line 228, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File
"/Users/karanalang/.conda/envs/PythonLeetcode/lib/python3.9/site-packages/pyspark/context.py",
line 384, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File
"/Users/karanalang/.conda/envs/PythonLeetcode/lib/python3.9/site-packages/pyspark/context.py",
line 144, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File
"/Users/karanalang/.conda/envs/PythonLeetcode/lib/python3.9/site-packages/pyspark/context.py",
line 331, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File
"/Users/karanalang/.conda/envs/PythonLeetcode/lib/python3.9/site-packages/pyspark/java_gateway.py",
line 108, in launch_gateway
    raise Exception("Java gateway process exited before sending its port
number")
Exception: Java gateway process exited before sending its port number

Mime
View raw message