quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Stein <gst...@lyra.org>
Subject Re: Session adventures
Date Wed, 30 Jul 2003 23:22:36 GMT
On Wed, Jul 30, 2003 at 05:30:06PM -0400, Gregory (Grisha) Trubetskoy wrote:
> APR provides an API for a global_mutex, but it had two things that I
> thought made it unusable for mod_python: (1) the lock has to be created in
> pre-fork stage, and (2) it uses a file per lock. With a potentially
> unlimited number of sessions, this would mean a lot of files.

The file-per-lock is *one* particular implementation. That isn't necessarily
the case on all platforms. In some cases, the lock might reside in shared
memory. In another case, it could be a global mutex on Windows.

Don't toss APR's locks just for that reason.

The "lock at startup" isn't a problem if you rejigger your thinking a little
bit. A little bit of redesign would fix things right up.

For example, consider creating one global lock to manage the session
management process. That global lock protects what you're going to do with
the rest of the data structures. In that structure, you have a mapping of
sessions currently being processed to one global lock from a pool. You can
then have one more global lock to cover any pool overflow.

The mapping provides a way to figure out which lock (from the pool) is being
used to arbitrate requests associated with a session. If too many
session-based-requests occur all at the same time, then that overflow will
all be gated by the single overflow lock. Presumably, the admin would set
the capacity properly, and watch for the log messages ("oh no! using the
overflow lock!"). Also recognize that many of the requests that the main
Apache server will be handling might not even it back to mod_python, and how
many of those will always be using sessions? In fact, you could probably
arrange the session management to only occur when a particular handle goes
to look at it. Many handlers, even though working for the same "user" or
"session" might not need access, so they won't have to acquire a lock to get
into the session object.

Anyways... this means that your pre-fork lock creation only needs to worry
about creating N locks, and be done with it.

> The best way to handle inter-process locking (with a potential for having
> thousands of locks) I could come up with was to use fcntl(2) file locking.

You could do this, but I'd recommend not. It is one of those shaky things
when it comes to portability. Then you have the issue about making sure it
doesn't go onto an NFS volume. Or other remoted mounts. etc etc.

I'd say to stick with APR and let it select the right mechanism.

[ fcntl probably *will* work just fine in most cases; it is always that edge
  case and needing to learn about them yet again ]

> So I was able to hack together a Python object which uses this
> fcntl+thread locking, which seems to work OK (given my very limited
> testing). It works something like:

Note that you can use ap_mpm_query() (or somesuch) and ask Apache what kind
of process/thread model it is using. You can fine-tune your approach based
on that. If it says "1 process, N threads", then you just use Python
threads. If it says "N processes, N threads", then you'll pull out the big
locking fun code...

> The second best option was using a dbm. It turns out many dbm libraries

Try using the apr_dbm_* functions from apr-util. Or you could directly use
the apr_sdbm_* functions. IIRC, those are even protected by a lock.

> that there isn't a free dbm that allows concurrent access (well, the
> implementation?). In any event, the instability went away when I tried
> gdbm.

Note that gdbm is GPL'd which means it might have a different definition of
"free" that you're intending. In particular, it is *NOT* LGPL'd which means
that it will impose GPL on mod_python if you don't make it optional.


Greg Stein, http://www.lyra.org/

View raw message