spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject error: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
Date Sat, 07 Aug 2021 18:08:25 GMT
Hi,


 I encounter the error:


"java.lang.UnsupportedOperationException: sun.misc.Unsafe or
java.nio.DirectByteBuffer.<init>(long, int) not available"


When reading from Google BigQuery (GBQ) table using Kubernetes cluster
built on debian buster


The current debian bustere from the docker image is:

root@ccf3ac45d0ed:/opt/spark/work-dir# cat /etc/*-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)"


And the Java version is


echo $JAVA_HOME

/usr/local/openjdk-11


Now according to Spark 3.1.2 doc <https://spark.apache.org/docs/latest/>


"*For Java 11*, -Dio.netty.tryReflectionSetAccessible=true is required
additionally for Apache Arrow library. This prevents
*java.lang.UnsupportedOperationException:
sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available* when
Apache Arrow uses Netty internally.


So I have used it as follows:

        spark-submit --verbose \

           --properties-file ${property_file} \

           --master k8s://https://$KUBERNETES_MASTER_IP:443 \

           --deploy-mode cluster \

           --name pytest \

           --conf
spark.yarn.appMasterEnv.PYSPARK_PYTHON=./pyspark_venv/bin/python \

           --py-files $CODE_DIRECTORY/DSBQ.zip \

           --conf spark.kubernetes.namespace=$NAMESPACE \

           --conf spark.executor.memory=5000m \

           --conf spark.network.timeout=300 \

           --conf spark.executor.instances=2 \

           --conf spark.kubernetes.driver.limit.cores=1 \

           --conf spark.driver.cores=1 \

           --conf spark.executor.cores=1 \

           --conf spark.executor.memory=2000m \

           --conf spark.kubernetes.driver.docker.image=${IMAGEGCP} \

           --conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \

           --conf spark.kubernetes.container.image=${IMAGEGCP} \

           --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark-bq \

          --conf
spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \

           --conf
spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"
\

           $CODE_DIRECTORY/${APPLICATION}

However, some comments mentioned
<https://stackoverflow.com/questions/62109276/errorjava-lang-unsupportedoperationexception-for-pyspark-pandas-udf-documenta>
that these parameters need to be supplied before spark-submit, so I added
them to $SPARK_HOME/conf/spark-defaults.conf


185@b272bbf663e6:/opt/spark/conf$ cat spark-defaults.conf

spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"

spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"

But I'm still getting the same error!


Any ideas will be appreciated.


Mich



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Mime
View raw message