spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <>
Subject Re: Improvement proposal | Dynamic disk allocation
Date Sun, 06 Nov 2016 13:13:31 GMT
If people agree that is desired, I am willing to submit a SIP for this and
find time to work on it.


On Sun, Nov 6, 2016 at 1:06 PM Aniket Bhatnagar <>

> Hello
> Dynamic allocation feature allows you to add executors and scale
> computation power. This is great, however, I feel like we also need a way
> to dynamically scale storage. Currently, if the disk is not able to hold
> the spilled/shuffle data, the job is aborted causing frustration and loss
> of time. In deployments like AWS EMR, it is possible to run an agent that
> add disks on the fly if it sees that the disks are running out of space and
> it would be great if Spark could immediately start using the added disks
> just as it does when new executors are added.
> Thanks,
> Aniket

View raw message