spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Yuqi" <Yuqi.Zh...@Teradata.com>
Subject [Spark Shell on AWS K8s Cluster]: Is there more documentation regarding how to run spark-shell on k8s cluster?
Date Mon, 29 Oct 2018 01:29:22 GMT
Hello guys,

I am Yuqi from Teradata Tokyo. Sorry to disturb but I have some problem regarding using spark
2.4 client mode function on kubernetes cluster, so I would like to ask if there is some solution
to my problem.

The problem is when I am trying to run spark-shell on kubernetes v1.11.3 cluster on AWS environment,
I couldn’t successfully run stateful set using the docker image built from spark 2.4. The
error message is showing below. The version I am using is spark v2.4.0-rc3.

Also, I wonder if there is more documentation on how to use client-mode or integrate spark-shell
on kubernetes cluster. From the documentation on https://github.com/apache/spark/blob/v2.4.0-rc3/docs/running-on-kubernetes.md
there is only a brief description. I understand it’s not the official released version yet,
but If there is some more documentation, could you please share with me?

Thank you very much for your help!


Error msg:
+ env
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ grep SPARK_JAVA_OPT_
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
--deploy-mode client "$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress= --deploy-mode
client
Error: Missing application resource.
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]


--
Yuqi Zhang
Software Engineer
m: 090-6725-6573


[signature_147554612]<http://www.teradata.com/>

2 Chome-2-23-1 Akasaka
Minato, Tokyo 107-0052
teradata.com<http://www.teradata.com>


This e-mail is from Teradata Corporation and may contain information that is confidential
or proprietary. If you are not the intended recipient, do not read, copy or distribute the
e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any
attachments. Thank you.

Please consider the environment before printing.


Mime
View raw message