qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiri Daněk (Jira) <j...@apache.org>
Subject [jira] [Commented] (PROTON-2111) python: memory leak on Container, SSL, and SSLDomain objects
Date Wed, 08 Apr 2020 15:35:00 GMT

    [ https://issues.apache.org/jira/browse/PROTON-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078390#comment-17078390
] 

Jiri Daněk commented on PROTON-2111:
------------------------------------

I am thinking that this is a duplicate of PROTON-2116. I tried the reproducer (using both
objgraph and gc.collect(); print(len(gc.get_objects())) and I observed that there is object
count growth even in proton 0.27, but it is nowhere as dramatic as in 0.28. Then I saw the
issue is fixed in 0.30, and it is also fixed in current master.

The test uses ssl. I think it is a good idea to have object leak test that uses ssl. I'll
clean up the test, decide whether to keep objgraph, use guppy (that is what I have some experience
with) or just the gc.get_objects, and send PR. For that reason I am not closing this issue
as duplicate just yet.

{noformat}
for v in 0.27.1 0.28.0 0.29.0 0.30.0; do                                                 
                                                                                         
                                 :(
        echo $v
        echo xxxxxxxxxxxxxxxxxxxxxx
        TESTDIR=qpid-proton/python/tests/integration
        INCLUDEDIR=qpid-proton/tests/py
        PYTHONPATH=qpid-proton_${v}/build/install/lib64/proton/bindings/python:${INCLUDEDIR}
\
          LD_LIBRARY_PATH=qpid-proton_${v}/build/install/lib64 \
            python -m unittest discover -v -s $TESTDIR -p '*_test.py'        
    done

0.27.1
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests)
... /home/jdanek/repos/qpid/qpid-proton/python/tests/integration/blocking_connection_fd_leak_test.py:148:
ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM,
proto=0, laddr=('::e092:8ad:e57f:0', 39407, 0, 4294967295)>
  urls = "amqps://{0}:{1}".format(*broker.get_acceptor_sockname())
ResourceWarning: Enable tracemalloc to get the object allocation traceback
amqps://:::39407
function                       8365     +8365
dict                           4234     +4234
tuple                          3157     +3157
list                           2154     +2154
weakref                        1927     +1927
builtin_function_or_method     1669     +1669
member_descriptor              1564     +1564
type                           1289     +1289
wrapper_descriptor             1284     +1284
getset_descriptor              1230     +1230
len gc get objects 32694
len gc get objects 32711
len gc get objects 32728
len gc get objects 32745
len gc get objects 32762
==============================================================================
Type                            Old_ids  Current_ids      New_ids Count_Deltas
==============================================================================
function                              0         8365        +8365        +8365
dict                                  0         4258        +4258        +4258
tuple                                 0         3151        +3151        +3151
list                                  0         2163        +2163        +2163
weakref                               0         1925        +1925        +1925
builtin_function_or_method            0         1667        +1667        +1667
member_descriptor                     0         1564        +1564        +1564
type                                  0         1289        +1289        +1289
wrapper_descriptor                    0         1284        +1284        +1284
getset_descriptor                     0         1230        +1230        +1230
==============================================================================
[<proton._transport.SSLDomain object at 0x7fe59f19e550>, <proton._transport.SSLDomain
object at 0x7fe59f19e7d0>]
[]
[<proton._transport.SSL object at 0x7fe59dbc5250>, <proton._transport.SSL object
at 0x7fe59dbc5390>, <proton._transport.SSL object at 0x7fe59dbc5550>, <proton._transport.SSL
object at 0x7fe59dbc5710>, <proton._transport.SSL object at 0x7fe59dbc58d0>]
ok

----------------------------------------------------------------------
Ran 1 test in 3.262s

OK
0.28.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests)
... amqps://127.0.0.1:37773
function                       8423     +8423
dict                           4250     +4250
tuple                          3178     +3178
list                           2158     +2158
weakref                        1935     +1935
builtin_function_or_method     1562     +1562
member_descriptor              1553     +1553
type                           1299     +1299
wrapper_descriptor             1284     +1284
getset_descriptor              1233     +1233
len gc get objects 32728
len gc get objects 32753
len gc get objects 32776
len gc get objects 32797
len gc get objects 32819
==============================================================================
Type                            Old_ids  Current_ids      New_ids Count_Deltas
==============================================================================
function                              0         8423        +8423        +8423
dict                                  0         4290        +4290        +4290
tuple                                 0         3178        +3178        +3178
list                                  0         2166        +2166        +2166
weakref                               0         1933        +1933        +1933
builtin_function_or_method            0         1560        +1560        +1560
member_descriptor                     0         1553        +1553        +1553
type                                  0         1299        +1299        +1299
wrapper_descriptor                    0         1284        +1284        +1284
getset_descriptor                     0         1233        +1233        +1233
==============================================================================
[<proton._transport.SSLDomain object at 0x7fb9a1069550>, <proton._transport.SSLDomain
object at 0x7fb9a10697d0>, <proton._transport.SSLDomain object at 0x7fb99f266790>,
<proton._transport.SSLDomain object at 0x7fb99f266ad0>, <proton._transport.SSLDomain
object at 0x7fb99f266350>, <proton._transport.SSLDomain object at 0x7fb99f266cd0>,
<proton._transport.SSLDomain object at 0x7fb99f266410>, <proton._transport.SSLDomain
object at 0x7fb99f266c90>, <proton._transport.SSLDomain object at 0x7fb99f266890>,
<proton._transport.SSLDomain object at 0x7fb99f2669d0>, <proton._transport.SSLDomain
object at 0x7fb99f266bd0>, <proton._transport.SSLDomain object at 0x7fb99f275890>,
<proton._transport.SSLDomain object at 0x7fb99fab6c10>, <proton._transport.SSLDomain
object at 0x7fb99fab69d0>]
[<proton._reactor.Container object at 0x7fb99facaf50>, <proton._reactor.Container
object at 0x7fb99f266a10>, <proton._reactor.Container object at 0x7fb99f266d50>,
<proton._reactor.Container object at 0x7fb99facae90>, <proton._reactor.Container
object at 0x7fb99fab7f10>, <proton._reactor.Container object at 0x7fb99faba0d0>]
[<proton._transport.SSL object at 0x7fb99f266c10>, <proton._transport.SSL object
at 0x7fb99f2750d0>, <proton._transport.SSL object at 0x7fb99f266d90>, <proton._transport.SSL
object at 0x7fb99f2752d0>, <proton._transport.SSL object at 0x7fb99f275210>, <proton._transport.SSL
object at 0x7fb99f275510>, <proton._transport.SSL object at 0x7fb99f275490>, <proton._transport.SSL
object at 0x7fb99f2757d0>, <proton._transport.SSL object at 0x7fb99f275610>, <proton._transport.SSL
object at 0x7fb99f275810>]
ok

----------------------------------------------------------------------
Ran 1 test in 0.600s

OK
0.29.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests)
... amqps://127.0.0.1:39825
function                       8423     +8423
dict                           4250     +4250
tuple                          3179     +3179
list                           2158     +2158
weakref                        1935     +1935
builtin_function_or_method     1562     +1562
member_descriptor              1553     +1553
type                           1299     +1299
wrapper_descriptor             1284     +1284
getset_descriptor              1233     +1233
len gc get objects 32721
len gc get objects 32746
len gc get objects 32778
len gc get objects 32798
len gc get objects 32826
==============================================================================
Type                            Old_ids  Current_ids      New_ids Count_Deltas
==============================================================================
function                              0         8423        +8423        +8423
dict                                  0         4290        +4290        +4290
tuple                                 0         3178        +3178        +3178
list                                  0         2166        +2166        +2166
weakref                               0         1933        +1933        +1933
builtin_function_or_method            0         1560        +1560        +1560
member_descriptor                     0         1553        +1553        +1553
type                                  0         1299        +1299        +1299
wrapper_descriptor                    0         1284        +1284        +1284
getset_descriptor                     0         1233        +1233        +1233
==============================================================================
[<proton._transport.SSLDomain object at 0x7f4278a4a550>, <proton._transport.SSLDomain
object at 0x7f4278a4a7d0>, <proton._transport.SSLDomain object at 0x7f4276c4cad0>,
<proton._transport.SSLDomain object at 0x7f4276c4cc50>, <proton._transport.SSLDomain
object at 0x7f4276c4cb50>, <proton._transport.SSLDomain object at 0x7f4276c4ca50>,
<proton._transport.SSLDomain object at 0x7f4276c4c610>, <proton._transport.SSLDomain
object at 0x7f4276c4cc10>, <proton._transport.SSLDomain object at 0x7f4276c4cd50>,
<proton._transport.SSLDomain object at 0x7f4277fcf990>, <proton._transport.SSLDomain
object at 0x7f4276c4c450>, <proton._transport.SSLDomain object at 0x7f4276c54950>,
<proton._transport.SSLDomain object at 0x7f427749c310>, <proton._transport.SSLDomain
object at 0x7f427749cb50>]
[<proton._reactor.Container object at 0x7f4276c4c950>, <proton._reactor.Container
object at 0x7f42774b3e90>, <proton._reactor.Container object at 0x7f427749c990>,
<proton._reactor.Container object at 0x7f427749c890>, <proton._reactor.Container
object at 0x7f42774b3110>, <proton._reactor.Container object at 0x7f427749eed0>]
[<proton._transport.SSL object at 0x7f4276c540d0>, <proton._transport.SSL object
at 0x7f4276c54250>, <proton._transport.SSL object at 0x7f4276c54090>, <proton._transport.SSL
object at 0x7f4276c54410>, <proton._transport.SSL object at 0x7f4276c544d0>, <proton._transport.SSL
object at 0x7f4276c4ca10>, <proton._transport.SSL object at 0x7f4276c54650>, <proton._transport.SSL
object at 0x7f4276c549d0>, <proton._transport.SSL object at 0x7f4276c54a10>, <proton._transport.SSL
object at 0x7f4276c54c90>]
ok

----------------------------------------------------------------------
Ran 1 test in 0.475s

OK
0.30.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests)
... amqps://127.0.0.1:33689
function                       8451     +8451
dict                           4255     +4255
tuple                          3200     +3200
list                           2158     +2158
weakref                        1939     +1939
builtin_function_or_method     1563     +1563
member_descriptor              1553     +1553
type                           1303     +1303
wrapper_descriptor             1284     +1284
getset_descriptor              1237     +1237
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
==============================================================================
Type                            Old_ids  Current_ids      New_ids Count_Deltas
==============================================================================
function                              0         8451        +8451        +8451
dict                                  0         4245        +4245        +4245
tuple                                 0         3198        +3198        +3198
list                                  0         2156        +2156        +2156
weakref                               0         1937        +1937        +1937
builtin_function_or_method            0         1561        +1561        +1561
member_descriptor                     0         1553        +1553        +1553
type                                  0         1303        +1303        +1303
wrapper_descriptor                    0         1284        +1284        +1284
getset_descriptor                     0         1237        +1237        +1237
==============================================================================
[<proton._transport.SSLDomain object at 0x7fc9f7658550>, <proton._transport.SSLDomain
object at 0x7fc9f76587d0>, <proton._transport.SSLDomain object at 0x7fc9f60bf850>,
<proton._transport.SSLDomain object at 0x7fc9f60bf610>]
[<proton._reactor.Container object at 0x7fc9f60bcf50>]
[]
ok

----------------------------------------------------------------------
Ran 1 test in 3.715s

OK
{noformat}

> python: memory leak on Container, SSL, and SSLDomain objects
> ------------------------------------------------------------
>
>                 Key: PROTON-2111
>                 URL: https://issues.apache.org/jira/browse/PROTON-2111
>             Project: Qpid Proton
>          Issue Type: Bug
>          Components: python-binding
>    Affects Versions: proton-c-0.28.0
>            Reporter: Chenxiong Qi
>            Assignee: Clifford Jansen
>            Priority: Major
>
> I have an application using qpid.proton to interact with ActiveMQ broker to publish messages.
Following is a fake script showing an example of the app works to send a message.
> {noformat}
> class SampleSender(proton.handlers.MessagingHandler):
>     def __init__(self, msg_id, *args, **kwargs):
>         super(SampleSender, self).__init__(*args, **kwargs)
>         self.msg_id = msg_id
>     def on_start(self, event):
>         ssl_domain = proton.SSLDomain(proton.SSLDomain.MODE_CLIENT)
>         ssl_domain.set_credentials(producer_config['certificate'],
>                                    producer_config['private_key'],
>                                    None)
>         ssl_domain.set_trusted_ca_db(producer_config['trusted_certificates'])
>         ssl_domain.set_peer_authentication(proton.SSLDomain.VERIFY_PEER)
>         conn = event.container.connect(urls=producer_config['urls'],
>                                        reconnect=False,
>                                        ssl_domain=ssl_domain)
>         event.container.create_sender(conn, target='topic://VirtualTopic.event')
>     def on_sendable(self, event):
>         msg = proton.Message(body={'msg-id': self.msg_id, 'name': 'python'})
>         event.sender.send(msg)
>         event.sender.close()
>         event.connection.close()
> def send_msg(msg_id):
>     container = proton.reactor.Container(SampleSender(msg_id))
>     container.run()
> objgraph.show_growth()
> for i in range(3):
>     send_msg(i + 1)
> new_ids = objgraph.get_new_ids()
> print(objgraph.at_addrs(new_ids['SSLDomain']))
> print(objgraph.at_addrs(new_ids['Container']))
> print(objgraph.at_addrs(new_ids['SSL']))
> {noformat}
> Each time to publish a message, a new Container object is created and a new handler object
is created and passed to that container. After the code runs several times, Container, SSL
and SSLDomain objects remain in memory and are not collected. I use objgraph to find out those
objects.
> I'm using Fedora 30 and python-qpid-proton 0.28.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message