quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Gallacher <...@jgassociates.ca>
Subject memory leaks (was Re: [jira] Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55)
Date Mon, 10 Jul 2006 12:17:57 GMT
I've run some tests to evaluate the memory leaks. The tests are brute
force - make requests and watch the memory changes with top -b.

These tests were run using Ubuntu 6.06, Apache 2.0.55 (mpm-worker with
StartServers = 2) and Python 2.4.3.

The baseline test is just a simple request to determine the initial
memory consumption and in theory should not leak. The memory footprint
is checked after each 1000 requests, and if greater than the memory
maximum the test is terminated and marked as FAIL. Tests with PASS
status may still be leaking, just not in a dramatic, fatal manner.

First up - our leaky 3.2.9, and man does it leak! The readlines request
has a body of 1M bytes, and it fails in a hurry.

============================================================
Mod_python Memory Leak Test
  mod_python version: 3.2.9
  requests attempted: 50000
  memory maximum for failure condition: 10.0%
============================================================
Test Name       Status   Requests      %Mem
------------------------------------------------------------
baseline         PASS      50000        1.1
cfgtree_walk     FAIL       5000       11.5
parse_qs         PASS      50000        2.2
parse_qsl        FAIL      15000       10.3
readlines        FAIL       1000       14.3


The following is the lastest from branches/3.2.x (includes backported
MODPYTHON-172 changes.)

============================================================
Mod_python Memory Leak Test
  mod_python version: 3.2.x build 20060709
  requests attempted: 50000
  memory maximum for failure condition: 10.0%
============================================================
Test Name       Status   Requests      %Mem
------------------------------------------------------------
baseline         PASS      50000        1.1
cfgtree_walk     PASS      50000        2.0
parse_qs         PASS      50000        2.2
parse_qsl        PASS      50000        2.1
readlines        PASS      50000        1.3


Everything is looking much better, with readlines looking very stable.
Cfgtree_walk and parse_qsl are also looking much better but there may
still be a problem. To confirm this I ran the tests against 3.2.x build
20060709 again, but with 100,000 requests.

============================================================
Mod_python Memory Leak Test
  mod_python version: 3.2.x build 20060709
  requests attempted: 100000
  memory maximum for failure condition: 10.0%
============================================================
Test Name       Status   Requests      %Mem
------------------------------------------------------------
baseline         PASS     100000        1.2
cfgtree_walk     PASS     100000        2.9
parse_qs         PASS     100000        3.0
parse_qsl        PASS     100000        3.1
readlines        PASS     100000        1.3


I'm still suspicious of cfgtree_walk, parse_qs and parse_qsl as their
memory consumption is creeping up. I've taken a quick look at the code
but I don't see any obvious problem. I'll re-run the tests tonight, but
with a much larger number of requests.

The next step is to look at other methods we should include for testing.
I would be nice to get 100% coverage, but I'm not sure that is really
practical. We should include the most commonly called methods as a minimum.

Jim

Jim Gallacher wrote:
> I'm running my leaktest suite against this commit now. Results to follow.
> 
> Jim
> 
> Nicolas Lehuen (JIRA) wrote:
>>     [ http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12419906
] 
>>
>> Nicolas Lehuen commented on MODPYTHON-172:
>> ------------------------------------------
>>
>> I've just backported the fix into the 3.2 branch.
>>
>>> Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
>>> --------------------------------------------------------------------------
>>>
>>>          Key: MODPYTHON-172
>>>          URL: http://issues.apache.org/jira/browse/MODPYTHON-172
>>>      Project: mod_python
>>>         Type: Bug
>>>   Components: core
>>>     Versions: 3.2.8
>>>  Environment: Win32 XP  SP1 / SP2
>>> Apache 2.0.55  installed from binary (.MSI)
>>> Python 2.4.2  or  2.4.3    installed from binary from www.python.org    
>>>     Reporter: Laurent Blanquet
>>>      Fix For: 3.3
>>> I encounter memory leaks [~ 16 K per request) using the configuration described
below.
>>> =============================
>>> Python configuration from Httpd.conf:
>>> =============================
>>> Alias /python/ "d:/python24/B2B/"      
>>> <Directory "d:/python24/B2B">
>>>         AddHandler mod_python .py                  
>>>         PythonHandler pyHandlerHTTP                  
>>>         PythonDebug On                             
>>> </Directory>                                       
>>> =============================
>>> Test handler -  pyHandlerHTTP.py :
>>> =============================
>>> import mod_python
>>> from mod_python import util
>>> def handler(req):
>>>       #Removing this line solves the problem.
>>>       F=util.FieldStorage( req )   
>>>       return mod_python.apache.OK
>>> =============================
>>> HTTP Request (dump using TCPWATCH):
>>> =============================
>>> POST http://localhost:80/python/Alertes.py HTTP/1.0
>>> Content-Type: multipart/form-data; boundary=--------061006144341906
>>> Content-Length: 209
>>> Proxy-Connection: keep-alive
>>> Host: www.tx2-localhost
>>> Accept: text/html, */*
>>> User-Agent: Mozilla/3.0 (compatible; Indy Library)
>>> Proxy-Authorization: Basic Og==
>>>  
>>> ----------061006144341906
>>> Content-Disposition: form-data; name="TYPE"
>>>  
>>> LAST_ALERTS
>>> ----------061006144341906
>>> Content-Disposition: form-data; name="FILEAGE"
>>>  
>>> 180
>>>  
>>> ----------061006144341906
> 
> 



Mime
View raw message