synapse-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Asankha C. Perera" <asan...@wso2.com>
Subject Re: Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
Date Sat, 24 Mar 2007 12:05:42 GMT
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Ant<br>
<br>
This is the same error seen by Indika on Windows.. and I think my
analysis is correct. If you run the test for the first time or after a
few minutes of running the test last, you should be able to go to
around 1000 iterations. After you start to hit this issue, even 200
iterations would give you the error. At this time, doing a netstat -na
should show you that most of the tcp ports are in TIME_WAIT state.
Usually it could take at least one minute till a port is cleared up by
the OS. The tuning parameters I specified for Linux tells the OS to use
the full port range for applications, and to set the tcp fin timeout to
30 secs - to clear up the ports as quickly as possible. Without *any*
OS tuning and on a Windows XP system - you definitely will encounter
this issue.<br>
<br>
asankha<br>
<br>
ant elder wrote:
<blockquote
 cite="mid71e1b5740703240356h34614dcaqcafd1e595dac59cd@mail.gmail.com"
 type="cite">I've tried again with the latest Synapse and HTTP
components code and several JVMs. The results feel slightly different
than before but the end result is still always the root exception
included below. Sometime it doesn't occur till around 1000 requests,
but sometimes it happens after not many requests at all.&nbsp; <br>
  <br>
&nbsp;&nbsp; ...ant<br>
  <br>
java.io.IOException: Unable to establish loopback connection<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.PipeImpl$Initializer.run(Unknown
Source)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.security.AccessController.doPrivileged(Native
Method)
  <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.PipeImpl.&lt;init&gt;(Unknown
Source)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown
Source)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.channels.Pipe.open(Unknown
Source)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.axis2.transport.nhttp.ServerHandler.requestReceived
(ServerHandler.java:108)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.axis2.transport.nhttp.LoggingNHttpServiceHandler.requestReceived(LoggingNHttpServiceHandler.java:83)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput
(DefaultNHttpServerConnection.java:96)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.axis2.transport.nhttp.PlainServerIOEventDispatch.inputReady(PlainServerIOEventDispatch.java:67)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.http.impl.nio.reactor.BaseIOReactor.readable
(BaseIOReactor.java:68)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:160)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java
:145)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:127)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java
:153)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.lang.Thread.run(Unknown
Source)<br>
Caused by: java.net.BindException: Address already in use: connect<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.Net.connect(Native
Method)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.SocketChannelImpl.connect
(Unknown Source)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.channels.SocketChannel.open(Unknown
Source)<br>
  <br>
  <div><span class="gmail_quote">On 3/23/07, <b
 class="gmail_sendername">Asankha C. Perera</b> &lt;<a
 href="mailto:asankha@wso2.com">
asankha@wso2.com</a>&gt; wrote:</span>
  <blockquote class="gmail_quote"
 style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:
1ex;">
    <div bgcolor="#ffffff" text="#000000">Ant<br>
    <br>
I am quite sure that the problem seen by Indika now was related to the
ports being exhausted - see the following articles and esp. the
"MaxUserPort" and "TcpTimedWaitDelay" parameters that could tweaked -
to be consistent with what I am using before running a load test on
Linux. I will ask Indika to check these on Monday - but you may try
this in the meantime if you get a chance<br>
    <br>
    <a
 href="http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx</a>
    <br>
    <a
 href="http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx
    </a><br>
    <a
 href="http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html
    </a>
    <div><span class="e" id="q_1117fd49330790b8_1"><br>
    <br>
asankha<br>
    <br>
Asankha C. Perera wrote:
    <blockquote cite="http://mid4603CD27.7080104@wso2.com" type="cite">
Hi Ant<br>
      <br>
I fixed this for Linux and JDK 1.5 - I am confident of this fix as I
was able to first recreate the issue consistently and then see the fix
in action using 5 concurrent users sending a total of 5000 messages
multiple times. However Indika is still seeing a 'similar' issue in
Windows using JDK 1.4. We will try to see if its related to JDK 1.4 or
Windows. If you get the latest nhttp code and build the nhttp JAR you
could verify this fix - and let me know.<br>
      <br>
I am listing some of the linux commands that came in handy for the
resolution incase someone wants to check this.<br>
      <br>
lsof -p 7426 =&gt; lists the open files for the pid given after the -p
option<br>
      <br>
ls -l /proc/9976/fd | wc -l =&gt; for each process the /proc filesystem
lists the files used and thus you could count the open files with this
command<br>
      <br>
asankha<br>
      <br>
Asankha C. Perera wrote:
      <blockquote cite="http://mid4603B078.1050500@wso2.com" type="cite">
Ant / Oleg<br>
        <br>
I can recreate this issue on both Windows and Linux and think its
caused by my code related to use of Pipes.. and I am actively looking
into this right now.. will get back to you on what I find.<br>
        <br>
asankha<br>
        <br>
ant elder wrote:
        <blockquote
 cite="http://mid71e1b5740703230151o35238107hac1b6806a9bbe23d@mail.gmail.com"
 type="cite">I've tried on several JDKs now and _always_ get similar
intermittent I/O related errors. I can use JMeter directly against
Axis2-1.1.1 without any problems at all, so this does look like some
issue with the NIO transport. Be really good to hear from other Windows
users to see if this is just my specific environment or&nbsp; a more general
problem problem. <br>
          <br>
To recreate:<br>
          <br>
1) build Synapse server sample by running 'ant' in the
samples\axis2Server\src\SimpleStockQuoteService directory<br>
2) start the sample service by running
samples\axis2Server\axis2server.bat <br>
3) get the Synapse config&nbsp; (either 8 or 501) from <a
 href="http://people.apache.org/%7Eantelder/temp/" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">http://people.apache.org/~antelder/temp/</a>,
put in repository\conf\sample and start syanps: bin\synapse.bat
-sample=8 <br>
4) get the JMeter config test1.jmx from <a
 href="http://people.apache.org/%7Eantelder/temp/" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">http://people.apache.org/~antelder/temp/</a>,
start Jmeter and File -&gt; Open and point to the test1.jmx file<br>
5) JMeter Run -&gt; Start and after not to long IO errors should appear
in the Syanpse console <br>
          <br>
&nbsp;&nbsp; ...ant <br>
          <br>
---------- Forwarded message ----------<br>
          <span class="gmail_quote">From: <b class="gmail_sendername">Asankha
C. Perera</b> &lt;<a href="mailto:asankha@wso2.com" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">asankha@wso2.com</a>&gt;<br>
Date: Mar 22, 2007 4:58 PM <br>
Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be
hanging because an unchecked exception caused the I/O dispatch thread
to terminate<br>
To: HttpComponents Project &lt;<a
 href="mailto:httpcomponents-dev@jakarta.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">
httpcomponents-dev@jakarta.apache.org</a>&gt;<br>
          <br>
          </span>
          <div bgcolor="#ffffff" text="#000000">Oleg/Ant <br>
          <br>
I am guessing this is something to do with Windows or the JDK you use..
But I am unable to test this week, so will try to my best to try this
sometime next week. As I said, on Linux I have run the system through
thousands of messages and multiple threads concurrently and have fixed
all the issues I came across.<br>
          <br>
So Oleg, I do not see this as a blocker for the <span name="st">HttpCore</span>
release - but
I will use your latest snapshots in Synapse to check on this in future
if it occurs again<br>
          <br>
thanks<br>
asankha<br>
          <br>
Oleg Kalnichevski (JIRA) wrote:
          <blockquote
 cite="http://mid10931418.1174568912176.JavaMail.jira@brutus"
 type="cite">
            <div><span>
            <pre>     [ <a
 href="https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

</a> ]

Oleg Kalnichevski resolved HTTPCORE-60.
---------------------------------------

    Resolution: Fixed

Anthony
It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O
dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the
Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl
to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.



Please review the changes and let me know if it is okay to proceed with the release

Oleg

  </pre>
            <blockquote type="cite">
              <pre>Transport appears to be hanging because an unchecked exception caused
the I/O dispatch thread to terminate
----------------------------------------------------------------------------------------------------------



                Key: HTTPCORE-60
                URL: <a
 href="https://issues.apache.org/jira/browse/HTTPCORE-60"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://issues.apache.org/jira/browse/HTTPCORE-60

</a>
            Project: HttpComponents Core
         Issue Type: Bug
   Affects Versions: 4.0-alpha4
           Reporter: ant elder
        Assigned To: Oleg Kalnichevski
            Fix For: 4.0-alpha4



See discussion on synapse-dev mailing list: <a
 href="http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html

</a>
The transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate. I believe there are several
different types of problems (at least two) that we are seeing here.

[I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
    </pre>
              <blockquote type="cite">
                <pre>java.nio.channels.ClosedChannelException
        at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
        at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java
:139)

      </pre>
              </blockquote>
            </blockquote>
            <pre>  </pre>
            </span></div>
          </blockquote>
          </div>
---------------------------------------------------------------------
To unsubscribe, e-mail: <a
 href="mailto:httpcomponents-dev-unsubscribe@jakarta.apache.org"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">httpcomponents-dev-unsubscribe@jakarta.apache.org</a>
For additional commands, e-mail: <a
 href="mailto:httpcomponents-dev-help@jakarta.apache.org"
 target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">httpcomponents-dev-help@jakarta.apache.org</a>
        </blockquote>
---------------------------------------------------------------------
To unsubscribe, e-mail: <a
 href="mailto:synapse-dev-unsubscribe@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-unsubscribe@ws.apache.org</a>
For additional commands, e-mail: <a
 href="mailto:synapse-dev-help@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-help@ws.apache.org</a>
      </blockquote>
---------------------------------------------------------------------
To unsubscribe, e-mail: <a
 href="mailto:synapse-dev-unsubscribe@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-unsubscribe@ws.apache.org</a>
For additional commands, e-mail: <a
 href="mailto:synapse-dev-help@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-help@ws.apache.org</a>
    </blockquote>
    </span></div>
    </div>
    <div><span class="e" id="q_1117fd49330790b8_3">---------------------------------------------------------------------
To unsubscribe, e-mail: <a
 href="mailto:synapse-dev-unsubscribe@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-unsubscribe@ws.apache.org</a>
For additional commands, e-mail: <a
 href="mailto:synapse-dev-help@ws.apache.org" target="_blank"
 onclick="return top.js.OpenExtLink(window,event,this)">synapse-dev-help@ws.apache.org</a>
    </span></div>
  </blockquote>
  </div>
  <br>
</blockquote>
</body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: synapse-dev-help@ws.apache.org


Mime
View raw message