spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hbogert <>
Subject Recreating the Mesos/Spark paper's experiments
Date Thu, 26 Mar 2015 22:56:18 GMT
Hi all, 

For my master thesis I will be characterising performance of two-level
schedulers like Mesos and after reading the paper:
where Spark is also introduced I am wondering how some experiments and
results came about. 
If this is not the place to ask these questions, or someone knows better
places, please let me know.

I am wondering if the experiment could show the same results if we would use
the current release of Spark, because in
the macro-benchmarks (Fig. 5c), we can see 4 instances (though the text
talks of of 5 instances) of Spark applications being run. 
During 1 instance Sparks seems to elastically grow especially between
[0,200] and [900,1100].

Already this would be problematic to recreate in current Spark on Mesos,
because once an application context starts, it 1) allocates 
all available nodes in the cluster and does not scale up or down during that
application’s lifetime in CoarseGrained mode — or 2) it 
allocates all memory, and does not release it, though it scales up and down
with regard to CPUs in FineGrained mode. 

Even in FineGrained mode it would not work well if there are other
frameworks who need a lot of memory, because they simply wouldn’t be able
to allocate it, because even during idle times of a spark application, the
cluster’s memory is taken. Of course we could limit the memory usage, but
this defeats the purpose of having Mesos.

Does someone know, 
1) Was there a memory limit for Spark during the experiments in the paper
(and thus was the nowadays FineGrained mode chosen), so that other
frameworks would also be able to run?
2) Was the Spark architecture vastly different back then?

Any other remarks, even anecdotal, are very welcome


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message