quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gregory (Grisha) Trubetskoy" <gri...@apache.org>
Subject Session adventures
Date Wed, 30 Jul 2003 21:30:06 GMT

For the past few days (weeks?) I've been experimenting with ways to
provide the much discussed session support in a most mod_python-esque way,
here are some notes:

Part I - locking

The first trick is session locking. At least JSP (haven't looked at
others) locks the session so that only one request with this particular
session can be processed at a time. This, of course, is a one-liner in
Java (just place a lock on the session object), but in the Apache
environment, which is a mixed multi-thread-process, this type of locking
is non trivial.

APR provides an API for a global_mutex, but it had two things that I
thought made it unusable for mod_python: (1) the lock has to be created in
pre-fork stage, and (2) it uses a file per lock. With a potentially
unlimited number of sessions, this would mean a lot of files.

The best way to handle inter-process locking (with a potential for having
thousands of locks) I could come up with was to use fcntl(2) file locking.
Fcntl allows one to lock a part of a file, and the offset to the locked
region can exceed the size of the file, so by locking a single byte we
could have an unlimited number of locks on a file of size 0. Fcntl, of
course, is not available on Windows, but there is no need for
inter-process locking on Windows since Apache runs as a single process.

Then we at the same time need thread locking as well. Again, the APR way
had some sort of a discarding limitation (I forgot what now), but Python
has a very decent cross-platform thread locking imlementation.

So I was able to hack together a Python object which uses this
fcntl+thread locking, which seems to work OK (given my very limited
testing). It works something like:

lock = apache.mutex()
lock.lock(1452375)
# do something
lock.unlock(1452375)

"1452375" becomes the offset into the file (for fcntl) and a key in a
dictionary of thread locks.

To come up with "1452375" I used hash(session_id). The session_id is a
string (I used an md5.hexdigest() to generate it, but I'd like to leave
other an option of generating their own way of generating a session id,
thus all we know about it is that it's a string). Using hash() to generate
the lock number can lead to same lock number resulting from different
session id's, but that isn't so much of a problem - there isn't too much
harm in two sessions sharing a lock, just a miniscule performance penalty.

Part II - session storage

The second trick is storing the session data across processes. Originally
I was thinking of using shared memory, and we have had a bit of discussion
on the subject, but that problem is just way too complicated. Perhaps POSH
folks will have something useful, but I wouldn't count on it for a while
(like a year)...

The second best option was using a dbm. It turns out many dbm libraries
mmap dbm files making them as fast as shared memory. I got pretty good
results with bsd dbm, the only drawback I discovered in the process is
that there isn't a free dbm that allows concurrent access (well, the
sleepycat one is supposed to, but it isn't free really). To overcome this
I had to place locks (using my crafty mutex) around every dbm access, as
well as having to open and close the file every time. The second major
problem was bsd db 1.85, which is just plain unstable (and this is on
FreeBSD, where it is heavily patched.... unless Python uses its own
implementation?). In any event, the instability went away when I tried
gdbm.

Part III - conclusion

I haven't checked any of this code in yet because it needs some cleaning,
but I did try a small un-scientific test by trying to take "Number Guess"
example that comes with Tomcat, and implementing it in mod_python. To my
surprise, even with all the dbm inefficiency, I could still beat Tomcat by
a factor of 3! (Of course I am Tomcat ignorant, and perhaps there is a way
to make it faster, plus this was JDK 1.3 (1.4 is supposed to be much
faster))

Signing off for now -

Grisha

Mime
View raw message