spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lian Jiang <>
Subject spark in jupyter cannot find a class in a jar
Date Thu, 15 Nov 2018 23:46:43 GMT
I am using spark in Jupyter as below:

import findspark

from pyspark import SQLContext, SparkContext
sqlCtx = SQLContext(sc)
df ="oci://mybucket@mytenant/myfile.parquet")

The error is:

Py4JJavaError: An error occurred while calling o198.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem
for scheme "oci"

I have put oci-hdfs-full- defining oci filesystem on all
namenodes and datanodes on hadoop.

export PYSPARK_SUBMIT_ARGS="--master yarn --deploy-mode client
pyspark-shell --driver-cores 8 --driver-memory 20g --num-executors 2
--executor-cores 6  --executor-memory 30g --jars
/mnt/data/hdfs/oci-hdfs-full- --conf
--conf spark.driver.extraClassPath=/mnt/data/hdfs/oci-hdfs-full-"

Any idea why this still happens? Thanks for any clue.

View raw message