quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Graham Dumpleton <grah...@dscpl.com.au>
Subject Re: memory leak in request.readline()
Date Sun, 13 Aug 2006 04:27:58 GMT
Where are we at with confirming these memory leaks and committing some
changes to fix them?

Only ask as I have a big patch related to MODPYTHON-63 that am wanting
to commit back into the repository, but don't want to be doing it if  
it is going
to make your search harder by introducing some new memory leaks. :-)

BTW, what platform does you leak tester require to run? On Mac OS X I am
always seeing slow memory leaks over time even with most basic handler.
This often makes it hard to know if changes I make are introducing new
memory leaks or not. :-(

Graham

On 12/08/2006, at 7:55 AM, Jim Gallacher wrote:

> Alexis Marrero wrote:
>> Where is the source code for _apache.make_table() ?
>
> It's a rather convoluted path, but src/tableobject.c is where you  
> end up.
>
> I've added a test to my leak test suite for apache.table().
> (apache.make_table() was deprecated at some point). It does indeed  
> leak
> like crazy.
>
> Jim
>
>> Alexis Marrero wrote:
>>> Jim,
>>>
>>> I found the culprit!!!
>>>
>>> There are two unrelated memory leaks.
>>>
>>> The first one is in req_readline().
>>>
>>> This code:
>>>
>>>    /* is there anything left in the rbuff from previous reads? */
>>>    if (self->rbuff_pos < self->rbuff_len) {
>>>              /* if yes, process that first */
>>>        while (self->rbuff_pos < self->rbuff_len) {
>>>            buffer[copied++] = self->rbuff[self->rbuff_pos];
>>>            if ((self->rbuff[self->rbuff_pos++] == '\n') ||
>>>                (copied == len)) {
>>>
>>>                /* our work is done */
>>>
>>>                /* resize if necessary */
>>>                if (copied < len)
>>>                    if(_PyString_Resize(&result, copied))
>>>                        return NULL;
>>>                return result;
>>>            }
>>>        }
>>>    }
>>>
>>> Should look like this:
>>>    /* is there anything left in the rbuff from previous reads? */
>>>    if (self->rbuff_pos < self->rbuff_len) {
>>>              /* if yes, process that first */
>>>        while (self->rbuff_pos < self->rbuff_len) {
>>>            buffer[copied++] = self->rbuff[self->rbuff_pos];
>>>            if ((self->rbuff[self->rbuff_pos++] == '\n') ||
>>>                (copied == len)) {
>>>
>>>                /* our work is done */
>>>
>>>                /* resize if necessary */
>>>                if (copied < len)
>>>                    if(_PyString_Resize(&result, copied))
>>>                        return NULL;
>>>                if (self->rbuff_pos >= self->rbuff_len && self->rbuff
>>> != NULL)
>>>                {
>>>                    free(self->rbuff);
>>>                    self->rbuff = NULL;
>>>                }
>>>                return result;
>>>            }
>>>        }
>>>    }
>>>
>>> That solves one.  Like I mentioned in one of the emails to the  
>>> mailing
>>> list, the buffer was not been freed in the last readline().
>>>
>>> The second one, for which I don't have a fix yet is
>>> apache.make_table() in mod_python/util.py line 152. If I comment  
>>> lines
>>> 152, 225, 227 you will see that memory doesn't grow.  I will keep
>>> investigating...
>>>
>>> Until the next email.
>>>
>>> /amn
>>> Jim Gallacher wrote:
>>>> I ran my baseline test with 500k requests, and got the following:
>>>> (Note that all the figures will have an error of +/- 0.1)
>>>>
>>>> baseline      500k requests     1.7%
>>>>
>>>>
>>>> So it would seem that there is not a specific problem in  
>>>> readline, or my
>>>> test case is messed up. FYI here are my 2 handlers:
>>>>
>>>> def baseline_handler(req):
>>>>     req.content_type = 'text/plain'
>>>>     req.write('ok baseline:')
>>>>     return apache.OK
>>>>
>>>>
>>>> def readline_handler(req):
>>>>     # the body of the request consists of
>>>>     # '\n'.join([ 'a'*10 for i in xrange(0,10)  ])
>>>>     req.content_type = 'text/plain'
>>>>     count = 0
>>>>     while(1):
>>>>         line = req.readline()
>>>>         if not line:
>>>>             break
>>>>         count += 1
>>>>
>>>>     req.write('ok readline: %d lines read' % count)
>>>>     return apache.OK
>>>>
>>>> Jim
>>>>
>>>>
>>>> Jim Gallacher wrote:
>>>>
>>>>> I'll have some time to investigate this over the next couple of  
>>>>> days. I
>>>>> ran my leaktest script for FieldStorage and readline, and  
>>>>> FieldStorage
>>>>> certainly still leaks, but I'm not so sure about readline itself.
>>>>>
>>>>> baseline      1k requests     1.2%
>>>>> readline    500k requests     1.6%
>>>>> fieldstorage    498k requests    10.1%
>>>>>
>>>>> The memory consumption figures are for a machine with 512MB ram.
>>>>>
>>>>> I'm running my baseline test with 500k requests right now to  
>>>>> see if the
>>>>> 1.6% figure for readline represents a real leak in that  
>>>>> function, or if
>>>>> it is just mod_python itself.
>>>>>
>>>>> My memory leak test suite is probably at the point that other  
>>>>> people
>>>>> will find it useful. Once I've written a README explaining its  
>>>>> use I'll
>>>>> commit it to the repository so everybody to play. If you anyone
>>>>> wants to
>>>>> give it a shot in the interim I can email it to you. Give me shout
>>>>> offlist.
>>>>>
>>>>> I haven't had a chance to look at the code you highlight below,  
>>>>> or at
>>>>> least not closely. The whole req_readline function looks like  
>>>>> it will
>>>>> require a good strong cup of coffee to fully comprehend. ;)
>>>>>
>>>>> Jim
>>>>>
>>>>> Alexis Marrero wrote:
>>>>>
>>>>>> Experimenting on this issue, I noticed that neither of the
>>>>>> following set
>>>>>> of "ifs" are ever met:
>>>>>>
>>>>>>
>>>>>>    786      /* Free old rbuff as the old contents have been  
>>>>>> copied
>>>>>> over and
>>>>>>    787         we are about to allocate a new rbuff. Perhaps this
>>>>>> could
>>>>>> be reused
>>>>>>    788         somehow? */
>>>>>>    789      if (self->rbuff_pos >= self->rbuff_len &&
self->rbuff
>>>>>> != NULL)
>>>>>>    790      {
>>>>>>    791          free(self->rbuff);
>>>>>>    792          self->rbuff = NULL;
>>>>>>    793      }
>>>>>>
>>>>>>
>>>>>> --------
>>>>>>
>>>>>>    846      /* Free rbuff if we're done with it */
>>>>>>    847      if (self->rbuff_pos >= self->rbuff_len &&
self->rbuff
>>>>>> != NULL)
>>>>>>    848      {
>>>>>>    849          free(self->rbuff);
>>>>>>    850          self->rbuff = NULL;
>>>>>>    851      }
>>>>>>
>>>>>> I noticed that by putting some statements to write to the output
>>>>>> stream.  They never execute.
>>>>>>
>>>>>> /amn
>>>>>>
>>>>>> On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
>>>>>>
>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> We are trying to nail down a memory leak that happens only when
>>>>>>> documents are POSTed to the server.
>>>>>>>
>>>>>>> For testing we have a short script that does:
>>>>>>>
>>>>>>> while True:
>>>>>>>     dictionary_of_parameters = {'field1': 'a'*100000}
>>>>>>>     post('url...', dictionary_of_parameters)
>>>>>>>
>>>>>>> Then we run "top" on the server and watch the server memory grow
>>>>>>> without bound.  Why do we know that the problem is in
>>>>>>> request.readline()?  If I go to
>>>>>>> mod_python.util.FieldStorage.read_to_boundary() and add the 

>>>>>>> following
>>>>>>> statement:
>>>>>>>
>>>>>>> def read_to_boundary(...):
>>>>>>>   return True
>>>>>>>   ...
>>>>>>>
>>>>>>> as the first executable line in the function the memory does
not
>>>>>>> grow.
>>>>>>>
>>>>>>> I have read the req_readline a 1000 time and I can't figure 

>>>>>>> out where
>>>>>>> the problem is.
>>>>>>>
>>>>>>>
>>>>>>> My config:
>>>>>>> Python 2.4.1
>>>>>>> mod_python 3.2.10
>>>>>>>
>>>>>>> Our request handler does nothing other than using
>>>>>>> util.FieldStorage(req) and req.write('hello').
>>>>>>>
>>>>>>> I have some suspicion that it has to do with:
>>>>>>> ....
>>>>>>>     19   * requestobject.c
>>>>>>>     20   *
>>>>>>>     21   * $Id: requestobject.c 420297 2006-07-09 13:53:06Z 

>>>>>>> nlehuen $
>>>>>>>     22   *
>>>>>>>     23   */
>>>>>>> ....
>>>>>>>    846      /* Free rbuff if we're done with it */
>>>>>>>    847      if (self->rbuff_pos >= self->rbuff_len &&
self- 
>>>>>>> >rbuff !=
>>>>>>> NULL)
>>>>>>>    848      {
>>>>>>>    849          free(self->rbuff);
>>>>>>>    850          self->rbuff = NULL;
>>>>>>>    851      }
>>>>>>>    852
>>>>>>>
>>>>>>> Though, I can't confirm.
>>>>>>>
>>>>>>>
>>>>>>> /amn
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>


Mime
View raw message