uima-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaroslaw Cwiklik <cwik...@apache.org>
Subject Re: [VOTE] UIMA-AS 2.10.2 RC5
Date Fri, 05 Jan 2018 16:29:19 GMT
Just to make this more clear, Eddie and I together determined that the AMQ
connection losses where due to
long GCs in the service. This conclusion was based on evidence found in the
service log and correlation with
jConsole stats.

Jerry

On Fri, Jan 5, 2018 at 10:56 AM, Jaroslaw Cwiklik <cwiklik@apache.org>
wrote:

> After reviewing the service log and looking at jConsole there are long GCs
> which cause connection loss and recovery.
> Looks like AE issue. Strange that these GCs happen if the service is idle.
> Someone should profile this AE to confirm
> where this and possibly fix the source of excessive garbage.
>
> Another possibility is to increase AMQ inactivityTimeout (default 30 secs)
> to prevent connection drops.
>
> Jerry
>
> On Tue, Jan 2, 2018 at 11:16 AM, Eddie Epstein <eaepstein@gmail.com>
> wrote:
>
>> There were no long GCs for this service. Note (from above) that many other
>> services running older versions of uima-as were also showing the same
>> behavior.
>>
>> Eddie
>>
>> On Tue, Jan 2, 2018 at 9:21 AM, Jaroslaw Cwiklik <cwiklik@apache.org>
>> wrote:
>>
>> > Looks like the client lost connection to the broker for some reason.
>> Long
>> > GC perhaps? But eventually the service should have recovered. Any
>> > indication of connection
>> > recovery later?
>> >
>> > Jerry
>> >
>> > On Sun, Dec 24, 2017 at 11:34 AM, Eddie Epstein <eaepstein@gmail.com>
>> > wrote:
>> >
>> > > Jerry,
>> > >
>> > > Testing a real analytic running 28 sync pipelines as a DUCC managed
>> > UIMA-AS
>> > > service, the DUCC pinger is reporting failures for an idle service 2
>> or 3
>> > > times a day.
>> > >
>> > > sm.log, an error surrounded by good pings, the ping normally done
>> every 2
>> > > minutes:
>> > >
>> > > Dec 2017 07:49:42,740  INFO SM.PingDriver - 10668 T[3588945]
>> > > handleStatistics  Ping ok:  UIMA-AS:eae-r1.16:tcp://broker1:61616
>> > > Alive[true] Healthy[true] + Info: QDEPTH[0] AveNQ[0] Consum[3] Prod[1]
>> > > minNQ[0] maxNQ[0] expCnt[0] inFlt[0] DQ[1] NQ[1] NDisp[1]
>> > > MetaNode[192.168.7.101] MetaPid[71061:91]
>> > > Dec 2017 07:51:42,844  INFO SM.PingDriver - 10668 T[3588945]
>> > > handleStatistics  Ping ok:  UIMA-AS:eae-r1.16:tcp://broker1:61616
>> > > Alive[true] Healthy[true] + Info: QDEPTH[0] AveNQ[0] Consum[3] Prod[1]
>> > > minNQ[0] maxNQ[0] expCnt[0] inFlt[0] DQ[1] NQ[1] NDisp[1]
>> > > MetaNode[192.168.7.101] MetaPid[71061:91]
>> > > Dec 2017 07:54:02,912 ERROR SM.PingDriver - 10668 T[3588945]
>> > > handleStatistics  Missed_pings  1 endpoint
>> > > UIMA-AS:eae-r1.16:tcp://broker1:61616 Alive[false] Healthy[false] +
>> > Info:
>> > > QDEPTH[0] AveNQ[0] Consum[3] Prod[1] minNQ[0] maxNQ[0] expCnt[0]
>> inFlt[0]
>> > > DQ[1] NQ[1] NDisp[1] MetaNode[N/A] MetaPid[N/A]: Cannot issue getMeta
>> to:
>> > > eae-r1.16:tcp://broker1:61616: java.util.concurrent.TimeoutException
>> > > Dec 2017 07:56:03,016  INFO SM.PingDriver - 10668 T[3588945]
>> > > handleStatistics  Ping ok:  UIMA-AS:eae-r1.16:tcp://broker1:61616
>> > > Alive[true] Healthy[true] + Info: QDEPTH[0] AveNQ[89570] Consum[3]
>> > Prod[1]
>> > > minNQ[89570] maxNQ[89570] expCnt[0] inFlt[0] DQ[1] NQ[1] NDisp[1]
>> > > MetaNode[192.168.7.101] MetaPid[71061:91]
>> > >
>> > > The broker logfile, complaining about all three service listeners
>> around
>> > > the same time (the service runs on 192.168.7.101):
>> > >
>> > > 2017-12-24 07:54:48,257 [yMonitor Worker] WARN  Transport
>> > >     - Transport Connection to: tcp://192.168.7.101:43895 failed:
>> > > org.apache.activemq.transport.InactivityIOException: Channel was
>> > inactive
>> > > for too (>30000) long: tcp://192.168.7.101:43895
>> > > 2017-12-24 07:54:48,287 [yMonitor Worker] WARN  Transport
>> > >     - Transport Connection to: tcp://192.168.7.101:43898 failed:
>> > > org.apache.activemq.transport.InactivityIOException: Channel was
>> > inactive
>> > > for too (>30000) long: tcp://192.168.7.101:43898
>> > > 2017-12-24 07:54:48,288 [yMonitor Worker] WARN  Transport
>> > >     - Transport Connection to: tcp://192.168.7.101:43899 failed:
>> > > org.apache.activemq.transport.InactivityIOException: Channel was
>> > inactive
>> > > for too (>30000) long: tcp://192.168.7.101:43899
>> > >
>> > > The service logfile also reporting problems for all three listeners:
>> > >
>> > > 07:55:12.457 - 91:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Service: Processing2 Runtime Exception
>> > > 07:55:12.457 - 91:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Jms Listener Failed. Endpoint: eae-r1.16 Managed By:
>> > > tcp://broker1:61616 Reason: javax.jms.JMSException:
>> java.io.EOFException
>> > > 07:55:12.457 - 91:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > ainer.
>> > > handleListenerSetupFailure:
>> > > WARNING: Uima AS Service:Processing2 Listener Unable To Connect To
>> > Broker:
>> > > tcp://broker1:61616 Retrying Until Successful ...
>> > > 07:55:12.457 - 2336:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Service: Processing2 Runtime Exception
>> > > 07:55:12.457 - 2336:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Jms Listener Failed. Endpoint: eae-r1.16 Managed By:
>> > > tcp://broker1:61616 Reason: javax.jms.JMSException:
>> java.io.EOFException
>> > > 07:55:12.457 - 2335:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Service: Processing2 Runtime Exception
>> > > 07:55:12.457 - 2335:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > > ainer.onException:
>> > > WARNING: Jms Listener Failed. Endpoint: eae-r1.16 Managed By:
>> > > tcp://broker1:61616 Reason: javax.jms.JMSException:
>> java.io.EOFException
>> > > 07:55:12.457 - 2336:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > ainer.
>> > > handleListenerSetupFailure:
>> > > WARNING: Uima AS Service:Processing2 Listener Unable To Connect To
>> > Broker:
>> > > tcp://broker1:61616 Retrying Until Successful ...
>> > > 07:55:12.457 - 2335:
>> > > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
>> > ainer.
>> > > handleListenerSetupFailure:
>> > > WARNING: Uima AS Service:Processing2 Listener Unable To Connect To
>> > Broker:
>> > > tcp://broker1:61616 Retrying Until Successful ...
>> > >
>> > >
>> > > sm.log is showing the same kind of ping failures for previous
>> versions of
>> > > UIMA-AS as well, not all at the same time as this event, but spread
>> out
>> > > widely. The clocks on all three machines are synchronized so it could
>> be
>> > > meaningful that the pinger client registered the error before the
>> broker
>> > or
>> > > the service.
>> > >
>> > > Don't think this is a blocker, just wanted to check if there is a
>> problem
>> > > in UIMA-AS.
>> > >
>> > > Eddie
>> > >
>> > >
>> > > On Thu, Dec 14, 2017 at 4:52 PM, Jaroslaw Cwiklik <cwiklik@apache.org
>> >
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > the UIMA-AS 2.10.2 release candidate 5 is ready for voting.
>> > > >
>> > > > This version contains the following fixes:
>> > > >  - fixed documentation
>> > > >
>> > > > The list of changes in Jira:
>> > > > http://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%
>> > > > 202.10.2AS%20AND%20project%20%3D%20UIMA
>> > > >
>> > > > The source and binary zip/tars are staged to
>> > > > http://dist.apache.org/repos/dist/dev/uima/uima-as/2.10.2/RC5/
>> > > >
>> > > > The eclipse update subsite is here:
>> > > > http://dist.apache.org/repos/dist/dev/uima/eclipse-update-
>> > site/uima-as/
>> > > >
>> > > > The Maven artifacts are here:
>> > > > http://repository.apache.org/content/repositories/orgapacheu
>> ima-1177
>> > > >
>> > > > The SVN tags are here:
>> > > > http://svn.apache.org/repos/asf/uima/uima-as/tags/uima-as-2.10.2/
>> > > >
>> > > > See http://uima.apache.org/testing-builds.html for suggestions on
>> how
>> > to
>> > > > test release candidates.
>> > > >
>> > > > Please vote on release:
>> > > >
>> > > > [ ] +1 OK to release
>> > > > [ ] 0   Don't care
>> > > > [ ] -1 Not OK to release, because ...
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message