hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests
Date Mon, 09 Feb 2015 22:21:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313004#comment-14313004
] 

Siddharth Seth commented on HADOOP-11552:
-----------------------------------------

Thanks for the feedback [~cmccabe].
bq.  Are you proposing to keep the TCP session open, but reuse the handler thread for something
else, while the RPC is progressing? It seems like this approach would alleviate the handler
thread bottleneck, but potentially run into another bottleneck by keeping open a lot of open
TCP sockets (file descriptors). Given that we've had problems with hitting max file descriptors
in the past, couldn't this be problematic?
Yes, the intent is to keep the TPC session open and re-use the handlers (decided by individual
protocol). That's a great point about FDs remaining open. Depending on where this mechanism
is used, I believe this could be controlled. Examples.
- AM-RM communication - one connection per AM running on a cluster. This is typically limited
to < #nodes on the cluster.
- NM-RM communication - if at all it makes sense to add it here. Would be limited to #nodes
on the cluster. (Similarly for DN-NN communication)
- Task-AM communication (MR as an example) - this definitely has the potential to blow up,
since the # of containers on a cluster can be large, and multiple AMs could be running on
the same node.

The approach doesn't force all protocols / all methods in a protocol to make use of this,
and only selective ones could be setup to use this.
It does, however, open up the option to design protocols without the #handler limit, where
applicable - and individual components would need to exercise this wisely.

bq. Heartbeats are fundamental to how YARN works. If we move to an offer-based system like
Mesos, it seems like the way to do it would be to have the resource manager make outgoing
connections to the executors, rather than keeping TCP sessions open a long time.
I don't think we need to make a drastic change to protocol like moving to the offer model
used in Mesos.
The request-response model used by YARN can be made faster though. Rightnow, heartbeats are
serving multiple purposes. Taking the AM-RM example, the allocate request serves as 1) a keep-alive
- this isn't required every second, and 2) request-response for container requests - the default
1 second heartbeat is to avoid delaying getting allocations for a previous request.
Full two-way communication would be ideal - where the RM opens connections when required,
but is a much larger change - and I doubt it'll be possible to make this in a compatible manner,
if we want to go down that route.

For individual components to use this - we could even consider adding new methods to make
use of this mechanism, and leave the old ones intact - with the option to use either mechanism.

> Allow handoff on the server side for RPC requests
> -------------------------------------------------
>
>                 Key: HADOOP-11552
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11552
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: HADOOP-11552.1.wip.txt
>
>
> An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal,
since this essentially implies that RPC operations should be short lived, and most operations
which could take time end up falling back to a polling mechanism.
> Some use cases where this is useful.
> - YARN submitApplication - which currently submits, followed by a poll to check if the
application is accepted while the submit operation is written out to storage. This can be
collapsed into a single call.
> - YARN allocate - requests and allocations use the same protocol. New allocations are
received via polling.
> The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'.
The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval.
awaitResponse is always left active with the RM - and returns the moment something is available.
> MapReduce/Tez task to AM communication is another example of this pattern.
> The same pattern of splitting calls can be used for other protocols as well. This should
serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat
can be sent less frequently.
> I believe there's some cases in HDFS as well, where the DN gets told to perform some
operations when they heartbeat into the NN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message