hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16403) Start a new statistical rpc queue and make the Reader's pendingConnection queue runtime-replaceable
Date Thu, 01 Aug 2019 11:57:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898009#comment-16898009
] 

Hadoop QA commented on HADOOP-16403:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 17s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 2 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 59s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 27s{color}
| {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 11m 42s{color} | {color:red}
root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 55s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 18s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 11s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  3s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 55s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 26s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 28s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 44s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 48s{color} | {color:red}
root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  0m 48s{color} | {color:red}
root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 48s{color} | {color:red}
root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 36s{color}
| {color:green} root: The patch generated 0 new + 544 unchanged - 11 fixed = 544 total (was
555) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 28s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 44s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  0s{color} | {color:red}
The patch has 53 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>.
Refer https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  2s{color} | {color:red}
The patch 400 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  1s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  0m 42s{color} | {color:red}
patch has errors when building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 24s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 31s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 56s{color} | {color:red}
hadoop-common-project_hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was
0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 24s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 42s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 28s{color} | {color:red}
The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 31s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16403 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12976420/HADOOP-16403.005.patch
|
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit
 shadedclient  findbugs  checkstyle  cc  xml  |
| uname | Linux 2670ad02a29c 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 89b102f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/branch-compile-root.txt
|
| findbugs | v3.1.0-RC1 |
| mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-compile-root.txt
|
| cc | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-compile-root.txt
|
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-compile-root.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/whitespace-eol.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/whitespace-tabs.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/testReport/ |
| asflicense | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/16442/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> Start a new statistical rpc queue and make the Reader's pendingConnection queue runtime-replaceable
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-16403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16403
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jinglun
>            Assignee: Jinglun
>            Priority: Major
>         Attachments: HADOOP-16403-How_MetricLinkedBlockingQueue_Works.pdf, HADOOP-16403.001.patch,
HADOOP-16403.002.patch, HADOOP-16403.003.patch, HADOOP-16403.004.patch, HADOOP-16403.005.patch,
MetricLinkedBlockingQueueTest.pdf
>
>
> I have an HA cluster with 2 NameNodes. The NameNode's meta is quite big so after the
active dead, it takes the standby more than 40s to become active. Many requests(tcp connect
request and rpc request) from Datanodes, clients and zkfc timed out and start retrying. The
suddenly request flood lasts for the next 2 minutes and finally all requests are either handled
or run out of retry times. 
>  Adjusting the rpc related settings might power the NameNode and solve this problem and
the key point is finding the bottle neck. The rpc server can be described as below:
> {noformat}
> Listener -> Readers' queues -> Readers -> callQueue -> Handlers{noformat}
> By sampling some failed clients, I find many of them got ConnectTimeoutException. It's
caused by a 20s un-responded tcp connect request. I think may be the reader queue is full
and block the listener from handling new connections. Both slow handlers and slow readers
can block the whole processing progress, and I need to know who it is. I think *a queue that
computes the qps, write log when the queue is full and could be replaced easily* will help.

>  I find the nice work HADOOP-10302 implementing a runtime-swapped queue. Using it at
Reader's queue makes the reader queue runtime-swapped automatically. The qps computing job
could be done by implementing a subclass of LinkedBlockQueue that does the computing job while
put/take/... happens. The qps data will show on jmx.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message