hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
Date Thu, 13 Feb 2014 16:48:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900494#comment-13900494

Daryn Sharp commented on HADOOP-10278:

I think some of the anomalies you are seeing may be due to other entropy.  I'd be concerned
if it was say a 5% difference, but a fraction of a percent isn't worth extra complexity and
locking.  I checked the source for take vs poll, and they are virtually identical sans one
calls condition.await() and the other condition.await(nanos).  Thus I don't see how poll time
impacts normal operation.  During a queue swap, a longer poll just means idle handlers waiting
on an empty queue will take a little longer to switch over to another queue - no big deal.
 I also can't see how poll time and client cpu can possibly be correlated.

Anyway, another RW lock or other synchronization to protect against the exceedingly rare,
if ever, use case of a queue swap is overkill.  Although the impact may appear minimal at
the scale you tested, I have grave concerns with 5-10 readers and 100-200 handlers.  Highly
contended locks exhibit very poor performance characteristics.

I glanced at the former pre-lock/custom-queue patch and I think it's pretty good.  Ie. bounded
take via poll and atomic ref swap.  It's minimally invasive and shouldn't pose any real-world
(non micro-benchmark) performance impact.  Drainto may need to be replaced to avoid undefined

I generally agree with Hiroshi on abstraction but creating a wrapper class to passthrough
all the methods just to convert take to poll may be overkill.

> Refactor to make CallQueue pluggable
> ------------------------------------
>                 Key: HADOOP-10278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10278
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: ipc
>            Reporter: Chris Li
>         Attachments: HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch
> * Refactor CallQueue into an interface, base, and default implementation that matches
today's behavior
> * Make the call queue impl configurable, keyed on port so that we minimize coupling

This message was sent by Atlassian JIRA

View raw message