spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Semenov <>
Subject Re: Sharing spark executor pool across multiple long running spark applications
Date Wed, 07 Feb 2018 14:37:48 GMT
The other way might be to launch a single SparkContext and then run jobs
inside of it.

You can take a look at these projects:

Problems with this way:
- Can't update the code of your job.
- A single job can break the SparkContext.

We evaluated this way and decided to go with the dynamic allocation,
but we also had to rethink the way we write our jobs:
- Can't use caching since it locks executors, have to use checkpointing,
which adds up to computation time.
- Use some unconventional methods like reusing the same DF to write out
multiple separate things in one go.
- Sometimes remove executors from within the job, like when we know how
many we would need, so the executors could join other jobs.

On Tue, Feb 6, 2018 at 3:00 PM, Nirav Patel <> wrote:

> Currently sparkContext and it's executor pool is not shareable. Each
> spakContext gets its own executor pool for entire life of an application.
> So what is the best ways to share cluster resources across multiple long
> running spark applications?
> Only one I see is spark dynamic allocation but it has high latency when it
> comes to real-time application.
> [image: What's New with Xactly] <>
> <>
> <>
> <>   <>
> <>

View raw message