spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Klaus Ma <klaus1982...@gmail.com>
Subject Re: Build customized resource manager
Date Mon, 11 Nov 2019 02:24:15 GMT
hm.... that'll be better to me if we can build customized resource manager
out of core; otherwise, we have to go through the long discussion in the
community :)
But if we support that, why still mesos/yarn/k8s resource manager there in
the tree?

On Fri, Nov 8, 2019 at 10:18 PM Tom Graves <tgraves_cs@yahoo.com> wrote:

> I don't know if it all works but some work was done to make cluster
> manager pluggable, see SPARK-13904.
>
> Tom
>
> On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma <
> klaus1982.cn@gmail.com> wrote:
>
>
> Any suggestions?
>
> - Klaus
>
> On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma <klaus1982.cn@gmail.com> wrote:
>
> Hi team,
>
> AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did
> some enhancement to them, e.g. integrate with Volcano
> <http://github.com/volcano-sh/volcano> in k8s. Is that possible to do
> that without fork the whole spark project? For example, enable customized
> resource manager with configuration, e.g. replace
> `org.apache.spark.deploy.k8s.submit.KubernetesClientApplication` with
> `MyK8SClient`, so I can only maintain the resource manager instead of the
> whole project.
>
> -- Klaus
>
>

Mime
View raw message