spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Gao <>
Subject [Spark on K8s] Scaling experiences sharing
Date Fri, 09 Nov 2018 16:26:17 GMT
Hi Spark Community,

I am reaching out to see if there are current large scale production or
pre-production deployment of Spark on k8s for batch and micro batch jobs.
Large scale means running 100s of thousand spark jobs daily and 1000s of
concurrent spark jobs on a single k8s cluster and 10s of millions of spark
executor pods daily (not concurrently).

If you happen to run and develop Spark on k8s at such scale, I'd want to
learn about your experience and scaling challenges and solutions.

Thank you,

View raw message