spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Federico D'Ambrosio" <>
Subject State of support for dynamic allocation on K8s and possible CMs
Date Mon, 01 Jul 2019 07:41:26 GMT
Hello everyone,

I wanted to ask what's the state of support of Spark dynamic allocation as
of now, if there's any issue where I could track its advancement and
missing features.

We've just started evaluating possible alternatives for a production
architectural setup for our use case, and dynamic allocation could be
useful since our processing batches have a moderate variance in terms of
number of processed objects during the lifetime of the application. Hence,
we'd like to see if K8s may fit as a Cluster Manager.

Our test environment is an Hadoop cluster (HDP 3.0, used because we had it
already around), but since Hadoop/HDFS is not a hard requirement, I'd like
to ask what's considered the best cluster manager: why should we use a
standalone cluster, wrt a YARN or MESOS cluster? I mean, obviously, if we
already had one of those clusters production-ready, the choice would be
easier, but starting from scratch what are the pros and cons of the various
spark-compatible alternatives.

Possibly, I'd like to ask if there's anyone who's had experience running
Spark on a public cloud (AWS, Azure etc.) and whether their experience
included Hadoop PaaS (such as EMR and HDInsight), full IaaS, any K8s aaS
(AKS, EKS etc.).

Thank you very much for your time,

View raw message