hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Albert Chu <ch...@llnl.gov>
Subject Re: Shuffling over the network for local map data.
Date Tue, 22 Jan 2013 19:42:07 GMT
I've experimented with similar changes in the hadoop trunk, although my
desire was to improve performance for networked file systems.  I had not
considered the idea that it could be used for files stored locally on

What type of performance tests did you run and what kind of improvements
did you find (or not find)?


On Tue, 2013-01-22 at 11:02 -0800, Suresh Kumar wrote:
> I have a patch that tries to use file links instead of making a copy
> of the data that is already available locally. I tested it on the a
> single machine cluster configuration running 48 mappers and reducers.
> I unfortunately do not have access to a cluster even a small one. Can
> some on review and test run my patch ?
> I created the patch using Eclipse against 1.0.3. My knowledge in Java
> in limited and the code is not well written in some classes. So please
> let me know if I need to make changes to the code along with a short
> explanation of the change.  I will happily do so. 
> Thanks,
> Suresh.
Albert Chu
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory

View raw message