quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Graham Dumpleton <grah...@dscpl.com.au>
Subject Re: two mod_python leaks
Date Thu, 19 Oct 2006 10:55:46 GMT
For such issues, by all means just send the email direct to the  
mailing list.
I have copied this back onto the list and I'll do some research into  
you describe.

BTW, I don't use Windows, but there are others on the list who do and it
may help if you can explain what methodology you are using to see the
Win32 handle leaks. This will help others to replicate it and then  
track it



On 19/10/2006, at 8:50 PM, Jeff Robbins wrote:

> Graham,
> I apologize in advance for not understanding the email etiquette  
> for the list, so I'm emailing you directly.  (I subscribed to the  
> python-dev list and email the faq magic subject but the reply said  
> "no info".)  Anyway...
> I know of two leaks in Python 3.2.10 / Apache 2.2.3 that I would  
> dearly love to get fixed.  One of them I have identified the source  
> and made a change to my copy and it works (and comports with the  
> python embedding documentation) so I was hoping you could either  
> get it in or help me understand how I can get it applied.
> The first leak is if you use the _repr_ function of a dict or list  
> (which is as simple as using str([]) in your mod_python handler!)   
> The repr functions use a dict in the python threadstate to avoid  
> some recursion problem.  This dict is allocated only on demand, so  
> while I found str([]) (or a str of any dict or list) to cause its  
> allocation, there may be other user code things you can do to get  
> it allocated.  To free it, the python embedder need to call  
> "PyThreadState_Clear" before calling "PyThreadState_Delete".  Yet  
> mod_python.c release_interpreter() does not call  
> "PyThreadState_Clear".  The fixed version of release_interpreter is  
> this:
> static void release_interpreter(void)
> {
> PyThreadState *tstate = PyThreadState_Get();
> #ifdef WITH_THREAD
> PyEval_ReleaseThread(tstate);
> #else
> PyThreadState_Swap(NULL);
> #endif
> // JSR
> PyThreadState_Clear(tstate);
> PyThreadState_Delete(tstate);
> }
> The Python embedded doc is clear that this is the correct thing to do:
> http://docs.python.org/api/threads.html
> ...
>        void PyThreadState_Clear( PyThreadState *tstate)
>  Reset all information in a thread state object. The interpreter  
> lock must be held.
>        void PyThreadState_Delete( PyThreadState *tstate)
>  Destroy a thread state object. The interpreter lock need not be  
> held. The thread state must have been reset with a previous call to  
> PyThreadState_Clear().
> ...
> The second leak, unfortunately, I have not been able to fix.  It is  
> in the parent httpd process on Win32 when you do a "restart" of  
> apache.  mod_python causes the leak of 4 Win32 Events (which are  
> likely allocated via use of various apr_ functions like the mutex  
> functions)   This is reproducible and easily visible in the windows  
> task manager if you look at the "handles" column and use either the  
> command line or the apache monitor to restart apache.  On my  
> installation, mod_python contributes 4 of the 5 leaking handles.   
> (Someone else leaks one also!!!)  This harms a long-term deployment  
> which uses the apache maxrequests feature to restart the child  
> every million requests.
> Thank you in advance for you patience with this long-winded email!
> Best regards,
> Jeff Robbins
> ----- Original Message ----- From: "Graham Dumpleton"  
> <grahamd@dscpl.com.au>
> To: <python-dev@httpd.apache.org>
> Sent: Wednesday, October 18, 2006 20:47
> Subject: Status of mod_python 3.3.
>> On JIRA, the following issues are still marked as incomplete for  
>> mod_python
>> version 3.3. I have noted my own comments about where they are up  
>> to and
>> what I think still needs to be done.
>> MODPYTHON-93 Improve util.FieldStorage efficiency.
>> This was actually marked as resolved but reopened because it was  
>> discovered
>> that changes meant that Trac <=0.9.6 would no longer work. The  
>> changes were
>> also backed out of mod_python 3.2.X branch and not released in  
>> 3.2.10.
>> At this point I believe we have agreed that code in 3.3 would be  
>> left as is and
>> people would need to use Trac >=0.10, which has now been release,  
>> with
>> mod_python 3.3 or later.
>> There was comments as to whether util.FieldStorage needs to have more
>> dictionary like access, but at this point I believe we should mark  
>> this issue
>> as resolved and if people want dictionary like access, they can  
>> open a
>> separate JIRA issue for that and we deal with it in a future release.
>> In summary, I believe we should mark this as resolved.
>> MODPYTHON-104 Allow Python code callouts with mod_include (SSI).
>> The code for all this has been done for some time. The only reason  
>> it hasn't
>> been marked as resolved as no documentation has been added into core
>> mod_python documentation. I have separately written a article on  
>> the new
>> feature which is available at:
>> http://www.dscpl.com.au/wiki/ModPython/Articles/ 
>> BasicsOfServerSideIncludes
>> I have no problem as this being used as basis for core documentation.
>> I had been holding off integrating it because of contention over  
>> whether we
>> could use wiki for documentation or not.
>> In summary, need to still keep this open until some documentation  
>> added
>> to core mod_python documentation.
>> MODPYTHON-127 Use namespace for mod_python PythonOption settings.
>> I have made code changes but not committed them back to  
>> repository. Jim
>> has committed some documentation changes related to it already  
>> though.
>> Some more documentation changes are probably required where  
>> options are
>> mentioned in relation to features they affect.
>> I had posed question about whether  
>> mod_python.session.database_directory
>> should also be added as a general fallback in cases where which  
>> type of
>> filesystem based session was not going to be known. Jim responded  
>> with +1,
>> but his explicit vs implicit comment made me unsure which proposal he
>> was agreeing with. Thus nothing done about that yet.
>> In summary, bit more work to do.
>> MODPYTHON-143 Implement and integrate a new module importer.
>> Code has been done, except for extra bit more logging of  
>> exceptions for when
>> modules hooks are called and a problem occurs. Also need to update  
>> the
>> documentation.
>> In summary, more work to do but mainly documentation.
>> MODPYTHON-186 Build process not using correct values from Python
>> config Makefile.
>> This is only known to be an issue on Mac OS X when the very latest
>> compiler tool chain software given out at Mac developer conference
>> is used. It may only affect new Intel Macs. Someone did suggest they
>> would come back with required changes but that hasn't happened.
>> Personally I would say we don't attempt to address this in 3.3 and  
>> defer
>> it till later.
>> MODPYTHON-190 Python 2.5 support.
>> From what I have seen, people are already using Python 2.5, thus  
>> is there
>> any urgency on this? All I can figure is that by not making  
>> changes you will
>> not be able to work with really large data, or will it all crash  
>> badly on a 64
>> bit platform with 64 bit support compiled in.
>> I don't know the answers to this and no one else (doesn't even  
>> have to be
>> one of the committers) has stepped up to do the work and work out  
>> what
>> the required changes are.
>> Personally I would say we don't attempt to address this in 3.3 and  
>> defer
>> it till later.
>> MODPYTHON-193 Add req.hlist.location to mirror req.hlist.directory.
>> I have done most of the code for this and now just sorting out some
>> problems with trailing slashes getting added when they shouldn't.  
>> Also
>> need to still update Session code and ensure None is returned for  
>> handler
>> directory when not used in a directory.
>> In summary, a bit more work to do.
>> So that is where we are at. Can I get some agreement that we will:
>> 1. Mark MODPYTHON-93 as resolved.
>> 2. Defer MODPYTHON-186 and MODPYTHON-190 till later.
>> Finally, is there anything else in outstanding issues (not listed  
>> here)
>> that people believe need to be address for mod_python 3.3?
>> Graham

View raw message