nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Date Mon, 13 Feb 2017 09:03:58 GMT
Hello,

The timeout happens when a node replicate a request to nodes to get a
merged cluster wide response, so increasing
nifi.cluster.node.read.timeout might help (in conf/nifi.properties).
Request replication shouldn't timeout ideally, increasing
nifi.web.jetty.threads might help to increase each node's capability
to handle more requests..

nifi.cluster.node.read.timeout=5 sec
nifi.web.jetty.threads=200

Would you share nifi thread dump result (./bin/nifi.sh dump, and
written in logs/nifi-bootstrap.log) while you are sending request from
NiFi UI and waiting its result arrives?
It might be helpful for us to find what causes the timeout or slowness.

Thanks,
Koji



On Mon, Feb 13, 2017 at 5:52 PM, Alessio Palma
<alessio.palma@docomodigital.com> wrote:
> Hello again,
> this is the stack trace which is raised on the connection timeout.
>
> 2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/status to mid1-a128-3.buongiorn
> o.com:8080 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
>         at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
>         at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
>         at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
>         at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
>         at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
>         at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
> ~[jersey-client-1.19.jar:1.19]
>         at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
>         at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
> ~[na:1.8.0_101]
>         at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_101]
>         at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_101]
>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_101]
>         at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> ~[na:1.8.0_101]
>         at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> ~[na:1.8.0_101]
>         at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_101]
>         at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> ~[na:1.8.0_101]
>         at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_101]
>         at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_101]
>         at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> ~[na:1.8.0_101]
>         at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_101]
>         at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> ~[na:1.8.0_101]
>         at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> ~[na:1.8.0_101]
>         at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_101]
>         at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> ~[na:1.8.0_101]
>         at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19]
>         at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19]
>         ... 12 common frames omitted
>
> perhaps it is useful to understand the problem.
>
>
>
> ________________________________
> From: Alessio Palma <alessio.palma@docomodigital.com>
> Sent: Monday, February 13, 2017 9:31:15 AM
> To: NIFI User mailing
> Subject: Cluster instability due to java.net.SocketTimeoutException: Read
> timed out
>
> Hello all,
> sometime the cluster is unstable due to "read time out".
> Is here any parameter which I can fine tuning in order to reduce this
> problem ?
> Most timeouts (66%) are coming from
>
> com.sun.jersey.api.client.ClientHandlerException
>
> few (33%) from
>
> o.apache.nifi.controller.FlowController
>
>
> Thanks in adavance
> AP
>

Mime
View raw message