hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyao Meng (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
Date Wed, 23 Oct 2019 07:35:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957628#comment-16957628

Siyao Meng commented on HADOOP-16656:

[~dineshchitlangia] Thanks for the comment.

Yes, in the most typical usage of FairCallQueue, {{[port_number]}} should be set to the NN
RPC port (in {{dfs.namenode.rpc-address}}). But I don't really see anything preventing users
from setting {{[port_number]}} to NN service RPC port (in dfs.namenode.servicerpc-address)
or lifeline port (in {{dfs.namenode.lifeline.rpc-address}}).

So I tested (trunk) locally with 1 NN and 1 DN,
(1) {{[port_number]}} can be set to NN RPC port (in {{dfs.namenode.rpc-address}}) *without*
setting a NN service RPC address/port ({{dfs.namenode.servicerpc-address}});
(2) {{[port_number]}} can be set to NN service RPC port (in {{dfs.namenode.servicerpc-address}});
(3) {{[port_number]}} can be set to NN lifeline RPC port (in {{dfs.namenode.lifeline.rpc-address}});

In all 3 cases above, I can start NN and DN, and successfully run a shell command "{{hdfs
dfs -ls /}}" with the NN.

One explanation is that, even if {{ipc.[port_number].callqueue.impl}} is *not* set, it defaults
to {{LinkedBlockingQueue}}, as shown in the code [here|https://github.com/apache/hadoop/blob/1654497f98fb7f2de8214d2fbad305b7a2854816/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L691].
So in theory, {{[port_number]}} could be set to any RPC port opened on NN.

I haven't dug into FairCallQueue implementation in its entirety - there might be some other
exceptions I didn't realize.

Instead, we could remind users that they typically want to set {{[port_number]}} to NN RPC
port rather than the other two (NN service RPC port, or lifeline port)?

> Document FairCallQueue configs in core-default.xml
> --------------------------------------------------
>                 Key: HADOOP-16656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16656
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>         Attachments: HADOOP-16656.001.patch
> So far those callqueue / scheduler / faircallqueue -related configurations are only documented
in FairCallQueue.md in 3.3.0:
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations
> (Thanks Akira for uploading this.)
> Goal: Document those configs in core-default.xml as well to make it easier for users(admins)
to find and use.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message