spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <>
Subject error: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
Date Sat, 07 Aug 2021 18:08:25 GMT

 I encounter the error:

"java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.<init>(long, int) not available"

When reading from Google BigQuery (GBQ) table using Kubernetes cluster
built on debian buster

The current debian bustere from the docker image is:

root@ccf3ac45d0ed:/opt/spark/work-dir# cat /etc/*-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)"

And the Java version is



Now according to Spark 3.1.2 doc <>

"*For Java 11*, -Dio.netty.tryReflectionSetAccessible=true is required
additionally for Apache Arrow library. This prevents
sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available* when
Apache Arrow uses Netty internally.

So I have used it as follows:

        spark-submit --verbose \

           --properties-file ${property_file} \

           --master k8s://https://$KUBERNETES_MASTER_IP:443 \

           --deploy-mode cluster \

           --name pytest \

spark.yarn.appMasterEnv.PYSPARK_PYTHON=./pyspark_venv/bin/python \

           --py-files $CODE_DIRECTORY/ \

           --conf spark.kubernetes.namespace=$NAMESPACE \

           --conf spark.executor.memory=5000m \

           --conf \

           --conf spark.executor.instances=2 \

           --conf spark.kubernetes.driver.limit.cores=1 \

           --conf spark.driver.cores=1 \

           --conf spark.executor.cores=1 \

           --conf spark.executor.memory=2000m \

           --conf spark.kubernetes.driver.docker.image=${IMAGEGCP} \

           --conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \

           --conf spark.kubernetes.container.image=${IMAGEGCP} \

spark.kubernetes.authenticate.driver.serviceAccountName=spark-bq \

spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \



However, some comments mentioned
that these parameters need to be supplied before spark-submit, so I added
them to $SPARK_HOME/conf/spark-defaults.conf

185@b272bbf663e6:/opt/spark/conf$ cat spark-defaults.conf



But I'm still getting the same error!

Any ideas will be appreciated.


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

View raw message