hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radim Kolar <...@filez.com>
Subject pluggable resources
Date Mon, 22 Oct 2012 10:41:10 GMT
I have proposal for improved resource scheduling.


as i see, development seems to go other way for example in 
https://issues.apache.org/jira/browse/YARN-2 for every added kind of 
resource there has to be significant rework.

you do not see benefits of having framework able to handle custom 
resource types? Its not all about memory and cores. You need to schedule 
jobs based on other factors (network capacity, availability of GPU 
cores, data locality).

And every cluster might have special considerations for example do not 
overload central SQL database. We usually have few hundred submitted 
jobs, proper resource sharing is essential. No point in running jobs 
which needs GPU which is in use by other mapper, better to run some 
other jobs until gpu becomes available again.

View raw message