hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Trezzo <ctre...@gmail.com>
Subject Review request: YARN-1492 (i.e. the YARN shared cache)
Date Fri, 05 Sep 2014 21:40:58 GMT
Hi All,

This email is to draw more attention to YARN-1492 and hopefully gain
traction on code reviews. At Twitter we have been running the shared cache
on our clusters and it has already served over 36 million requests.

The new shared cache feature is relatively isolated from existing code and
is completely enabled/disabled by configuration. When disabled there are no
behavioral changes compared to the existing code base. It would be great to
see this committed to trunk and even more awesome if it makes 2.6.

This is a larger patch, but I have broken it up into a number of sub-tasks
in an attempt to make it more digestible for the review process. A couple
things to note:

1. The two patches that interact with existing code in a substantial way
YARN-2236 <https://issues.apache.org/jira/browse/YARN-2236>
<https://issues.apache.org/jira/browse/MAPREDUCE-5951> - This patch adds
the cache uploader service to the node manager. MAPREDUCE-5951
<https://issues.apache.org/jira/browse/MAPREDUCE-5951> - This patch adds
support for the shared cache at the MapReduce layer allowing jobs to cache
job jars, lib jars, files and archives.

2. If you would like to try out the entire feature there is a "big bang"
patch in YARN-1492 and instructions on how to set it up in a comment on the
issue here

Please ping me if you have any questions or think there is something else I
could do to make reviewing easier.

Chris Trezzo

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message