spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: [Spark Shell on AWS K8s Cluster]: Is there more documentation regarding how to run spark-shell on k8s cluster?
Date Wed, 31 Oct 2018 09:34:07 GMT
Just out of curiosity why would you not use Glue (which is Spark on
kubernetes) or EMR?

Regards,
Gourav Sengupta

On Mon, Oct 29, 2018 at 1:29 AM Zhang, Yuqi <Yuqi.Zhang@teradata.com> wrote:

> Hello guys,
>
>
>
> I am Yuqi from Teradata Tokyo. Sorry to disturb but I have some problem
> regarding using spark 2.4 client mode function on kubernetes cluster, so I
> would like to ask if there is some solution to my problem.
>
>
>
> The problem is when I am trying to run spark-shell on kubernetes v1.11.3
> cluster on AWS environment, I couldn’t successfully run stateful set using
> the docker image built from spark 2.4. The error message is showing below.
> The version I am using is spark v2.4.0-rc3.
>
>
>
> Also, I wonder if there is more documentation on how to use client-mode or
> integrate spark-shell on kubernetes cluster. From the documentation on
> https://github.com/apache/spark/blob/v2.4.0-rc3/docs/running-on-kubernetes.md
> there is only a brief description. I understand it’s not the official
> released version yet, but If there is some more documentation, could you
> please share with me?
>
>
>
> Thank you very much for your help!
>
>
>
>
>
> Error msg:
>
> + env
>
> + sed 's/[^=]*=\(.*\)/\1/g'
>
> + sort -t_ -k4 -n
>
> + grep SPARK_JAVA_OPT_
>
> + readarray -t SPARK_EXECUTOR_JAVA_OPTS
>
> + '[' -n '' ']'
>
> + '[' -n '' ']'
>
> + PYSPARK_ARGS=
>
> + '[' -n '' ']'
>
> + R_ARGS=
>
> + '[' -n '' ']'
>
> + '[' '' == 2 ']'
>
> + '[' '' == 3 ']'
>
> + case "$SPARK_K8S_CMD" in
>
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
>
> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress= --deploy-mode client
>
> Error: Missing application resource.
>
> Usage: spark-submit [options] <app jar | python file | R file> [app
> arguments]
>
> Usage: spark-submit --kill [submission ID] --master [spark://...]
>
> Usage: spark-submit --status [submission ID] --master [spark://...]
>
> Usage: spark-submit run-example [options] example-class [example args]
>
>
>
>
>
> --
>
> Yuqi Zhang
>
> Software Engineer
>
> m: 090-6725-6573
>
>
> [image: signature_147554612] <http://www.teradata.com/>
>
> 2 Chome-2-23-1 Akasaka
>
> Minato, Tokyo 107-0052
> teradata.com <http://www.teradata.com>
>
>
> This e-mail is from Teradata Corporation and may contain information that
> is confidential or proprietary. If you are not the intended recipient, do
> not read, copy or distribute the e-mail or any attachments. Instead, please
> notify the sender and delete the e-mail and any attachments. Thank you.
>
> Please consider the environment before printing.
>
>
>
>
>

Mime
View raw message