quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Gallacher <...@jgassociates.ca>
Subject Re: memory leak in request.readline()
Date Fri, 11 Aug 2006 21:55:35 GMT
Alexis Marrero wrote:
> Where is the source code for _apache.make_table() ?

It's a rather convoluted path, but src/tableobject.c is where you end up.

I've added a test to my leak test suite for apache.table().
(apache.make_table() was deprecated at some point). It does indeed leak
like crazy.

Jim

> Alexis Marrero wrote:
>> Jim,
>>
>> I found the culprit!!!
>>
>> There are two unrelated memory leaks.
>>
>> The first one is in req_readline().
>>
>> This code:
>>
>>    /* is there anything left in the rbuff from previous reads? */
>>    if (self->rbuff_pos < self->rbuff_len) {
>>              /* if yes, process that first */
>>        while (self->rbuff_pos < self->rbuff_len) {
>>            buffer[copied++] = self->rbuff[self->rbuff_pos];
>>            if ((self->rbuff[self->rbuff_pos++] == '\n') ||
>>                (copied == len)) {
>>
>>                /* our work is done */
>>
>>                /* resize if necessary */
>>                if (copied < len)
>>                    if(_PyString_Resize(&result, copied))
>>                        return NULL;
>>                return result;
>>            }
>>        }
>>    }
>>
>> Should look like this:
>>    /* is there anything left in the rbuff from previous reads? */
>>    if (self->rbuff_pos < self->rbuff_len) {
>>              /* if yes, process that first */
>>        while (self->rbuff_pos < self->rbuff_len) {
>>            buffer[copied++] = self->rbuff[self->rbuff_pos];
>>            if ((self->rbuff[self->rbuff_pos++] == '\n') ||
>>                (copied == len)) {
>>
>>                /* our work is done */
>>
>>                /* resize if necessary */
>>                if (copied < len)
>>                    if(_PyString_Resize(&result, copied))
>>                        return NULL;
>>                if (self->rbuff_pos >= self->rbuff_len && self->rbuff
>> != NULL)
>>                {
>>                    free(self->rbuff);
>>                    self->rbuff = NULL;
>>                }
>>                return result;
>>            }
>>        }
>>    }
>>
>> That solves one.  Like I mentioned in one of the emails to the mailing
>> list, the buffer was not been freed in the last readline().
>>
>> The second one, for which I don't have a fix yet is
>> apache.make_table() in mod_python/util.py line 152. If I comment lines
>> 152, 225, 227 you will see that memory doesn't grow.  I will keep
>> investigating...
>>
>> Until the next email.
>>
>> /amn
>> Jim Gallacher wrote:
>>> I ran my baseline test with 500k requests, and got the following:
>>> (Note that all the figures will have an error of +/- 0.1)
>>>
>>> baseline      500k requests     1.7%
>>>
>>>
>>> So it would seem that there is not a specific problem in readline, or my
>>> test case is messed up. FYI here are my 2 handlers:
>>>
>>> def baseline_handler(req):
>>>     req.content_type = 'text/plain'
>>>     req.write('ok baseline:')
>>>     return apache.OK
>>>
>>>
>>> def readline_handler(req):
>>>     # the body of the request consists of
>>>     # '\n'.join([ 'a'*10 for i in xrange(0,10)  ])
>>>     req.content_type = 'text/plain'
>>>     count = 0
>>>     while(1):
>>>         line = req.readline()
>>>         if not line:
>>>             break
>>>         count += 1
>>>
>>>     req.write('ok readline: %d lines read' % count)
>>>     return apache.OK
>>>
>>> Jim
>>>
>>>
>>> Jim Gallacher wrote:
>>>  
>>>> I'll have some time to investigate this over the next couple of days. I
>>>> ran my leaktest script for FieldStorage and readline, and FieldStorage
>>>> certainly still leaks, but I'm not so sure about readline itself.
>>>>
>>>> baseline      1k requests     1.2%
>>>> readline    500k requests     1.6%
>>>> fieldstorage    498k requests    10.1%
>>>>
>>>> The memory consumption figures are for a machine with 512MB ram.
>>>>
>>>> I'm running my baseline test with 500k requests right now to see if the
>>>> 1.6% figure for readline represents a real leak in that function, or if
>>>> it is just mod_python itself.
>>>>
>>>> My memory leak test suite is probably at the point that other people
>>>> will find it useful. Once I've written a README explaining its use I'll
>>>> commit it to the repository so everybody to play. If you anyone
>>>> wants to
>>>> give it a shot in the interim I can email it to you. Give me shout
>>>> offlist.
>>>>
>>>> I haven't had a chance to look at the code you highlight below, or at
>>>> least not closely. The whole req_readline function looks like it will
>>>> require a good strong cup of coffee to fully comprehend. ;)
>>>>
>>>> Jim
>>>>
>>>> Alexis Marrero wrote:
>>>>   
>>>>> Experimenting on this issue, I noticed that neither of the
>>>>> following set
>>>>> of "ifs" are ever met:
>>>>>
>>>>>
>>>>>    786      /* Free old rbuff as the old contents have been copied
>>>>> over and
>>>>>    787         we are about to allocate a new rbuff. Perhaps this
>>>>> could
>>>>> be reused
>>>>>    788         somehow? */
>>>>>    789      if (self->rbuff_pos >= self->rbuff_len &&
self->rbuff
>>>>> != NULL)
>>>>>    790      {
>>>>>    791          free(self->rbuff);
>>>>>    792          self->rbuff = NULL;
>>>>>    793      }
>>>>>
>>>>>
>>>>> --------
>>>>>
>>>>>    846      /* Free rbuff if we're done with it */
>>>>>    847      if (self->rbuff_pos >= self->rbuff_len &&
self->rbuff
>>>>> != NULL)
>>>>>    848      {
>>>>>    849          free(self->rbuff);
>>>>>    850          self->rbuff = NULL;
>>>>>    851      }
>>>>>
>>>>> I noticed that by putting some statements to write to the output
>>>>> stream.  They never execute.
>>>>>
>>>>> /amn
>>>>>
>>>>> On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
>>>>>
>>>>>     
>>>>>> All,
>>>>>>
>>>>>> We are trying to nail down a memory leak that happens only when
>>>>>> documents are POSTed to the server.
>>>>>>
>>>>>> For testing we have a short script that does:
>>>>>>
>>>>>> while True:
>>>>>>     dictionary_of_parameters = {'field1': 'a'*100000}
>>>>>>     post('url...', dictionary_of_parameters)
>>>>>>
>>>>>> Then we run "top" on the server and watch the server memory grow
>>>>>> without bound.  Why do we know that the problem is in
>>>>>> request.readline()?  If I go to
>>>>>> mod_python.util.FieldStorage.read_to_boundary() and add the following
>>>>>> statement:
>>>>>>
>>>>>> def read_to_boundary(...):
>>>>>>   return True
>>>>>>   ...
>>>>>>
>>>>>> as the first executable line in the function the memory does not
>>>>>> grow.
>>>>>>
>>>>>> I have read the req_readline a 1000 time and I can't figure out where
>>>>>> the problem is.
>>>>>>
>>>>>>
>>>>>> My config:
>>>>>> Python 2.4.1
>>>>>> mod_python 3.2.10
>>>>>>
>>>>>> Our request handler does nothing other than using
>>>>>> util.FieldStorage(req) and req.write('hello').
>>>>>>
>>>>>> I have some suspicion that it has to do with:
>>>>>> ....
>>>>>>     19   * requestobject.c
>>>>>>     20   *
>>>>>>     21   * $Id: requestobject.c 420297 2006-07-09 13:53:06Z nlehuen
$
>>>>>>     22   *
>>>>>>     23   */
>>>>>> ....
>>>>>>    846      /* Free rbuff if we're done with it */
>>>>>>    847      if (self->rbuff_pos >= self->rbuff_len &&
self->rbuff !=
>>>>>> NULL)
>>>>>>    848      {
>>>>>>    849          free(self->rbuff);
>>>>>>    850          self->rbuff = NULL;
>>>>>>    851      }
>>>>>>    852
>>>>>>
>>>>>> Though, I can't confirm.
>>>>>>
>>>>>>
>>>>>> /amn
>>>>>>
>>>>>>         
>>>>     
>>>
>>>   
>>
> 
> 


Mime
View raw message