spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artemis User <arte...@dtechspace.com>
Subject Re: error: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
Date Sat, 07 Aug 2021 19:02:50 GMT
Without seeing the code and the whole stack trace, just a wild guess if 
you set the config param for enabling arrow 
(spark.sql.execution.arrow.pyspark.enabled)?  If not in your code, you 
would have to set it in the spark-default.conf.   Please note that the 
parameter spark.sql.execution.arrow.enabled is deprecated since Spark 3.0...

-- ND

On 8/7/21 2:08 PM, Mich Talebzadeh wrote:
>
> Hi,
>
>
>  I encounter the error:
>
>
> "java.lang.UnsupportedOperationException: sun.misc.Unsafe or 
> java.nio.DirectByteBuffer.<init>(long, int) not available"
>
>
> When reading from Google BigQuery (GBQ) table using Kubernetes cluster 
> built on debian buster
>
>
> The current debian bustere from the docker image is:
>
> root@ccf3ac45d0ed:/opt/spark/work-dir# cat /etc/*-release
>
> PRETTY_NAME="Debian GNU/Linux 10 (buster)"
>
>
> And the Java version is
>
>
> echo $JAVA_HOME
>
> /usr/local/openjdk-11
>
>
> Now according to Spark 3.1.2 doc <https://spark.apache.org/docs/latest/>
>
>
> "_For Java 11_, |-Dio.netty.tryReflectionSetAccessible=true| is 
> required additionally for Apache Arrow library. This prevents 
> |/java.lang.UnsupportedOperationException: sun.misc.Unsafe or 
> java.nio.DirectByteBuffer.(long, int) not available/| when Apache 
> Arrow uses Netty internally.
>
>
> So I have used it as follows:
>
> spark-submit --verbose \
>
>  --properties-file ${property_file} \
>
>  --master k8s://https://$KUBERNETES_MASTER_IP:443 \
>
>  --deploy-mode cluster \
>
>  --name pytest \
>
>  --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./pyspark_venv/bin/python \
>
>  --py-files $CODE_DIRECTORY/DSBQ.zip \
>
>  --conf spark.kubernetes.namespace=$NAMESPACE \
>
>  --conf spark.executor.memory=5000m \
>
>  --conf spark.network.timeout=300 \
>
>  --conf spark.executor.instances=2 \
>
>  --conf spark.kubernetes.driver.limit.cores=1 \
>
>  --conf spark.driver.cores=1 \
>
>  --conf spark.executor.cores=1 \
>
>  --conf spark.executor.memory=2000m \
>
>  --conf spark.kubernetes.driver.docker.image=${IMAGEGCP} \
>
>  --conf spark.kubernetes.executor.docker.image=${IMAGEGCP} \
>
>  --conf spark.kubernetes.container.image=${IMAGEGCP} \
>
>  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-bq \
>
>  --conf 
> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" 
> \
>
>  --conf 
> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" 
> \
>
>  $CODE_DIRECTORY/${APPLICATION}
>
>
> However, some comments mentioned 
> <https://stackoverflow.com/questions/62109276/errorjava-lang-unsupportedoperationexception-for-pyspark-pandas-udf-documenta>

> that these parameters need to be supplied before spark-submit, so I 
> added them to $SPARK_HOME/conf/spark-defaults.conf
>
>
> 185@b272bbf663e6:/opt/spark/conf$ cat spark-defaults.conf
>
> spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"
>
> spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true"
>
>
> But I'm still getting the same error!
>
>
> Any ideas will be appreciated.
>
>
> Mich
>
> *Disclaimer:* Use it at your own risk.Any and all responsibility for 
> any loss, damage or destruction of data or any other property which 
> may arise from relying on this email's technical content is explicitly 
> disclaimed. The author will in no case be liable for any monetary 
> damages arising from such loss, damage or destruction.
>


Mime
View raw message