spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <>
Subject How sensitive is Spark to Swap?
Date Mon, 07 Nov 2016 17:28:59 GMT
This may seem like a silly question, but it really isn’t. 
In terms of Map/Reduce, its possible to over subscribe the cluster because there is a lack
of sensitivity if the servers swap memory to disk. 

In terms of HBase, which is very sensitive, swap doesn’t just kill performance, but also
can kill HBase. (I’m sure one can tune it to be less sensitive…) 

But I have to ask how sensitive is Spark? 
Considering we can cache to disk (local disk) it would imply that it would less sensitive.

Yet we see some posters facing over subscription and hitting OOME. 


To unsubscribe e-mail:

View raw message