quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Graham Dumpleton (JIRA)" <j...@apache.org>
Subject [jira] Created: (MODPYTHON-241) Incremental memory use when performing streaming over long period.
Date Tue, 07 Aug 2007 07:04:03 GMT
Incremental memory use when performing streaming over long period.
------------------------------------------------------------------

                 Key: MODPYTHON-241
                 URL: https://issues.apache.org/jira/browse/MODPYTHON-241
             Project: mod_python
          Issue Type: Bug
          Components: core
    Affects Versions: 3.3.1, 3.2.10
            Reporter: Graham Dumpleton
            Priority: Minor


As first raised in:

  http://www.modpython.org/pipermail/mod_python/2007-July/023964.html

when streaming data from mod_python handler over a long period of time, memory use will keep
increasing.

Note that this isn't actually a memory leak though, as the memory gets reclaimed at the end
of the request.

The underlying problem arises from mod_python using the ap_rwrite() function to write output
data. When this function is used it will create a bucket chain for each block of data written,
with the bucket chain coming from the request memory pool. There is no attempt to reuse the
bucket chain for successive writes, so memory will increase on every call.

The normal solution to this problem is to retain the bucket chain and use it from one call
to the next. In mod_python though this isn't simple to achieve because calls to response data
may or may not be flushed at the discretion of the user code. The retaining of the bucket
chain and using it from one call to the next generally can only be done when data is always
being flushed. A more complicated system would be needed which can cope with a need to buffer
data at times and yet flush at other times.

One approach may be to not actually use Apache itself to buffer data between flushes, but
to use Python data structures instead to accumulate references to Python string objects in
a Python list so objects are still valid. When flushed only then construct a chain of buckets
which use direct reference to raw data of Python string objects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message