qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Moravec (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (QPID-6297) Python client (qpid.messaging) raises KeyError insead of reconnecting
Date Sun, 11 Jan 2015 15:49:34 GMT

    [ https://issues.apache.org/jira/browse/QPID-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272940#comment-14272940
] 

Pavel Moravec commented on QPID-6297:
-------------------------------------

Backtrace in human-readable form:

File "/usr/lib/python2.7/site-packages/gofer/transport/qpid/consumer.py", line 116, in get
return self.__receiver.fetch(timeout=timeout) 
File "<string>", line 6, in fetch 
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1041, in fetch self._ecwait(lambda:
not self.draining) 
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait result
= self._ewait(lambda: self.closed or predicate(), timeout) 
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 993, in _ewait result
= self.session._ewait(lambda: self.error or predicate(), timeout) 
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 580, in _ewait result
= self.connection._ewait(lambda: self.error or predicate(), timeout) 
File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait self.check_error()

File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 212, in check_error
raise e InternalError: Traceback (most recent call last): 
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 663, in write op.dispatch(self)

File "/usr/lib/python2.7/site-packages/qpid/ops.py", line 84, in dispatch getattr(target,
handler)(self, *args) 
File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 888, in do_session_detached
sst = self._sessions.pop(dtc.channel) KeyError: 0

Potential cause:
1) client calls receiver.fetch with "high" timeout (here 10seconds) - no msg available, library
waiting to broker or timeout
2) library detects connection drop, so it detaches the session (with traceback:
[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py',
811, '__bootstrap_inner', 'self.run()'), ('/usr/lib64/python2.7/threading.py', 764, 'run',
'self.__target(*self.__args, **self.__kwargs)'), ('/usr/lib/python2.7/site-packages/qpid/selector.py',
141, 'run', 'sel.readable()'), ('<string>', 6, 'readable', None), ('/usr/lib/python2.7/site-packages/qpid/messaging/driver.py',
422, 'readable', 'self.engine.write(data)'), ('/usr/lib/python2.7/site-packages/qpid/messaging/driver.py',
664, 'write', 'op.dispatch(self)'), ('/usr/lib/python2.7/site-packages/qpid/ops.py', 84, 'dispatch',
'getattr(target, handler)(self, *args)'), ('/usr/lib/python2.7/site-packages/qpid/messaging/driver.py',
886, 'do_session_detached', 'sss = "removing dtc.channel=" + str(dtc.channel) + "\\n" + str(traceback.extract_stack())
+ "\\n"')]
)

3) on the fetch timeout, internal exception is raised about session detached, so the connection
driver is asked for removing the session (while it has been removed)


This should have trivial reproducer (I hope), something like:

qpid-receive.py --timeout=10 -a "testQueue; {create:always}" -m10

and blocking iptables after a while (receiver should cycle)

(will test it later on)

> Python client (qpid.messaging) raises KeyError insead of reconnecting
> ---------------------------------------------------------------------
>
>                 Key: QPID-6297
>                 URL: https://issues.apache.org/jira/browse/QPID-6297
>             Project: Qpid
>          Issue Type: Bug
>          Components: Python Client
>    Affects Versions: 0.22
>         Environment: EL6
>            Reporter: Jeff Ortel
>         Attachments: goferBug.cap
>
>
> Description of problem:
> Having some temporary network outage causing gofer loses TCP connection to AMQP broker,
it does not try to reconnect.
> How reproducible:
> 100%
> Steps to Reproduce:
> 1. Just to speedup reproducer, lower kernel tunable net.ipv4.tcp_retries2 to e.g. 4:
> echo 4 > /proc/sys/net/ipv4/tcp_retries2
> 2. Have consumer connected (with auto-reconnect enabled and heartbeats not enabled) and
receiver open on a queue address and check its TCP connections to AMQP broker:
> netstat -anp | grep 5671
> (there should be 2 TCP connections)
> 3. Emulate network outage via iptables:
> iptables -A OUTPUT -p tcp --dport 5671 -j REJECT
> 4. Monitor /var/log/messages; once it logs WARNING "recoverable error", flush iptables
(iptables -F).
> 5. Wait few seconds.
> 6. Check gofer TCP connections:
> netstat -anp | grep 5671
> Actual results:
> 6. shows just 1 TCP connection
> /var/log/messages repeatedly logs:
> Dec  1 16:39:02 pmoravec-rhel6-3 goferd: [ERROR][pulp.agent.a726580c-5f1e-4a79-9f11-de0adc52c1e9]
gofer.transport.qpid.consumer:117 - 046d2084-b0f1-4de4-a039-89499d9e680d
> Dec  1 16:39:02 pmoravec-rhel6-3 goferd: [ERROR][pulp.agent.a726580c-5f1e-4a79-9f11-de0adc52c1e9]
gofer.transport.qpid.consumer:117 - Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/gofer/transport/qpid/consumer.py",
line 113, in get return self.__receiver.fetch(timeout=timeout) File "<string>", line
6, in fetch File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 1030,
in fetch self._ecwait(lambda: self.linked) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py",
line 50, in _ecwait result = self._ewait(lambda: self.closed or predicate(), timeout) File
"/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 993, in _ewait result
= self.session._ewait(lambda: self.error or predicate(), timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py",
line 580, in _ewait result = self.connection._ewait(lambda: self.error or predicate(), timeout)
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait self.check_error()
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in check_error
raise e InternalError: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py",
line 660, in write op.dispatch(self) File "/usr/lib/python2.6/site-packages/qpid/ops.py",
line 84, in dispatch getattr(target, handler)(self, *args) File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py",
line 877, in do_session_detached sst = self._sessions.pop(dtc.channel) KeyError: 'pop(): dictionary
is empty'
> Expected results:
> 2nd TCP connection re-established, no errors in /var/log/messages



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message