crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Rose (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-602) Combiner initialization repeatedly retrieves RT nodes from DistCache, leading to high NN load
Date Wed, 20 Apr 2016 16:06:25 GMT


Michael Rose commented on CRUNCH-602:

At least in our environment that seems to be the case. Our NN was on really weak hardware
and it consistently hit 100% CPU while jobs with combiners were running, which could just
be basic metadata RPC traffic. I'm not sure if it was grabbing the contents each time. In
any case, it would be better to avoid the M*R network calls during map phase.

I think I'll have time today to reduce debug DistCache call down a bit and see which RPCs
the filesystem calls are sending and if it's possible to cache those, given that the distributed
cache shouldn't be changing mid-job.

> Combiner initialization repeatedly retrieves RT nodes from DistCache, leading to high
NN load
> ---------------------------------------------------------------------------------------------
>                 Key: CRUNCH-602
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.12.0, 0.13.0, 0.14.0
>         Environment: Crunch 0.14-SNAPSHOT, CDH5.6.0
>            Reporter: Michael Rose
>            Assignee: Josh Wills
>              Labels: performance
>         Attachments: crunch-602.patch
> When running one of our Crunch pipelines, we noticed our NameNode under very heavy load.
We run our masters on pretty light hardware, so our NN was sitting at 100% CPU.
> Crunch reads the RTNodes during creation of a CrunchTaskContext. These are created when
Mappers and Reducers are created. Importantly, a CrunchCombiner is a subclass of a Reducer,
so each mapper will create R combiners where R is the number of reducers and thus R CrunchTaskContexts.
Consequently in highly parallel jobs, this means M*R semi-expensive calls to the NameNode.
> In the constructor for CrunchTaskContext, this is the read to the DistCache:
> this.nodes = (List<RTNode>), path);
> Which then leads to a read into the NN + deserialization.
> For now, we took the overly simplistic approach of caching the results of the DistCache
read in a Guava cache. The cache ensures combiners reuse RTNodes with only the overhead of
deserialization which is somewhat unavoidable as RTNodes are stateful and not reusable. However,
it's not configurable except by modifying code.
> I'll attach the patch, but given that it's not yet configurable I wouldn't call it a
"fix available." There may be much better ways of fixing this issue as well -- if you have
some guidance I'd be happy to do the legwork on a patch.

This message was sent by Atlassian JIRA

View raw message