spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <aniket.bhatna...@gmail.com>
Subject Improvement proposal | Dynamic disk allocation
Date Sun, 06 Nov 2016 13:06:55 GMT
Hello

Dynamic allocation feature allows you to add executors and scale
computation power. This is great, however, I feel like we also need a way
to dynamically scale storage. Currently, if the disk is not able to hold
the spilled/shuffle data, the job is aborted causing frustration and loss
of time. In deployments like AWS EMR, it is possible to run an agent that
add disks on the fly if it sees that the disks are running out of space and
it would be great if Spark could immediately start using the added disks
just as it does when new executors are added.

Thanks,
Aniket

Mime
View raw message