spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hsu, Philip" <>
Subject Issue with Running Spark in Jupyter Notebook
Date Thu, 24 Jun 2021 07:08:14 GMT
Hi there,

My name is Philip, a master’s student at Imperial College London. I’m trying to use Spark
to complete my course work assignment. I ran the following code:

from pyspark import SparkContext
sc = SparkContext.getOrCreate()

and got the following error message:

Py4JJavaError: An error occurred while calling
: java.lang.NoClassDefFoundError: Could not initialize class org.sparkproject.jetty.http.MimeTypes
        at org.sparkproject.jetty.server.handler.gzip.GzipHandler.<init>(
        at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:485)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:147)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:147)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:147)
        at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:486)
        at org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:486)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:486)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(
        at java.base/java.lang.reflect.Constructor.newInstance(
        at py4j.reflection.MethodInvoker.invoke(
        at py4j.reflection.ReflectionEngine.invoke(
        at py4j.Gateway.invoke(
        at py4j.commands.ConstructorCommand.invokeConstructor(
        at py4j.commands.ConstructorCommand.execute(
        at java.base/

While in my Macbook’s terminal, it’s showing following error messages:

WARNING: An illegal reflective access operation has occurred

pyspark_mongodb_nb  | WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform
(file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor

pyspark_mongodb_nb  | WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform

pyspark_mongodb_nb  | WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations

pyspark_mongodb_nb  | WARNING: All illegal access operations will be denied in a future release

pyspark_mongodb_nb  | 21/06/24 06:57:17 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

pyspark_mongodb_nb  | Using Spark's default log4j profile: org/apache/spark/

pyspark_mongodb_nb  | Setting default log level to "WARN".

pyspark_mongodb_nb  | To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use

pyspark_mongodb_nb  | 21/06/24 06:57:20 WARN MacAddressUtil: Failed to find a usable hardware
address from the network interfaces; using random bytes: bd:af:a7:b4:a2:46:2a:28

I’m wondering if you could help me resolve the issues I have with my laptop. I have a 2020
MacBook Pro with a M1 chip. Thank you so much in advance.


Philip Hsu

View raw message