spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Garlapati, Suryanarayana (Nokia - IN/Bangalore)" <suryanarayana.garlap...@nokia.com>
Subject RE: External shuffle service on K8S
Date Sat, 27 Oct 2018 08:11:00 GMT
Hi,
There is an unmerged PR which can be used against spark 2.4(if you are interested) or master
branch(3.0). Spark 2.3 K8S lacks lot of features. I suggest you upgrade to 2.4 which will
be released in few days from now.

https://github.com/apache/spark/pull/22722

Regards
Surya

From: Matt Cheah <mcheah@palantir.com>
Sent: Saturday, October 27, 2018 6:12 AM
To: Li Gao <eyesofhorus@gmail.com>; vincent.gromakowski@gmail.com
Cc: caolijun1166@gmail.com; user@spark.apache.org
Subject: Re: External shuffle service on K8S

Hi there,

Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this
matter.

-Matt Cheah

From: Li Gao <eyesofhorus@gmail.com<mailto:eyesofhorus@gmail.com>>
Date: Friday, October 26, 2018 at 9:10 AM
To: "vincent.gromakowski@gmail.com<mailto:vincent.gromakowski@gmail.com>" <vincent.gromakowski@gmail.com<mailto:vincent.gromakowski@gmail.com>>
Cc: "caolijun1166@gmail.com<mailto:caolijun1166@gmail.com>" <caolijun1166@gmail.com<mailto:caolijun1166@gmail.com>>,
"user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: External shuffle service on K8S

There are existing 2.2 based ext shuffle on the fork:
https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html [apache-spark-on-k8s.github.io]<https://urldefense.proofpoint.com/v2/url?u=https-3A__apache-2Dspark-2Don-2Dk8s.github.io_userdocs_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=enaZSDsU2MW_A_HY6l9TvqhhqoKOMWjbPvEOngFz1oA&e=>

You can modify it to suit your needs.

-Li


On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <vincent.gromakowski@gmail.com<mailto:vincent.gromakowski@gmail.com>>
wrote:
No it's on the roadmap >2.4

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <caolijun1166@gmail.com<mailto:caolijun1166@gmail.com>>
a écrit :
Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes?

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html
[spark.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=1uVGe6F4jUe4jMJnoKyMPlbr4qanZg4AL4vVWuwMjaY&e=>),
but couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao


Mime
View raw message