quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gregory (Grisha) Trubetskoy" <gri...@apache.org>
Subject Re: [mod_python] Using shared memory to do global persistence
Date Fri, 11 Jul 2003 02:12:26 GMT

Well, I've been able to rig something together to have a hash table in
shared memory that is capable of storing simple strings, which is a start.
I haven't checked any of this in because I'm having doubts about whether
this is the way to go.

I used the hash table implementation that is part of mod_ssl
(ssl_util_table.c), because that was the simplest way I could think of to
get this going.

But that file actually contains "(C) Gray Watson <gray@letters.com>", and
his own license underneath the Apache one - anyone know the story behind
this, and whether this is going to be a problem? Should I ask this on some
other list?

Also, I'm not very sure what the implications would be if mod_python and
mod_ssl are loaded at the same time. I think it should be OK, but I wonder
if there are situations in which there would be some sort of a
name-conflict problem, and therefore the functions in it better be
prepended with our own prefix.

Other options I looked at:

1. Something using or similar to POSH. This would rule, but it's too much
work. Posh also relies on it having the cotrol over forking(), which won't
work in our situation (i think).

2. Using an APR hash table - impossible because it uses apr_palloc()
familiy and does not provide a way to allocate memory via a custom func.
It'd be nice if APR had a shared memory table, I might ask about this on
the APR list.

Grisha


On Tue, 8 Jul 2003, Gregory (Grisha) Trubetskoy wrote:

>
> Thanks for the tip on rmm. I've been looking at the code and mod_ssl seems
> to use it extensively, in fact mod_ssl has a usable implementation of a
> shared memory hash table used for session tracking, even with timeouts..
>
> I am not very clear why mod_ssl uses rmm, other than as a way to allocate
> and free blocks of memory within a large shared memory segment.
>
> E.g. its hash table (which, btw is not an APR hash table), which is able
> to use pluggable malloc funcs (those in turn use rmm), still stores real
> pointers, and to free memory mod_ssl does stuff like [slightly
> simplified]:
>
>     apr_rmm_off_t off = apr_rmm_offset_get(RMM, ptr);
>     apr_rmm_free(RMM, off);
>
> Either I'm missing something, or converting a real pointer (possibly
> originally created in a different process) to an offset via
> apr_rmm_offset_get() is not a safe thing to do if access to rmm was
> obtained via apr_rmm_attach(), as opposed to apr_rmm_init(). (Mod_ssl does
> not need to use apr_rmm_attach(), which is why it works I think)
>
> I suppose for our purposes, the same strategy (doing rmm_init at module
> initialization/pre-fork time) might work, though it would be nice if we
> could use an APR hash table that stored rmm offsets (I haven't looked at
> that yet, but the fact that mod_ssl uses a home-grown table implementation
> suggests that it may not be possible) this way shared memory could be
> created as-needed from within child processes.
>
> Grisha
>
> On Thu, 26 Jun 2003, Greg Stein wrote:
>
> > APR has facilities to do shared memory in a portable fashion; APR is part of
> > Apache 2.0, so there isn't much reason to go grab any other library.
> >
> > You can also use APRUTIL's apr_rmm.h to manage sub-allocations within the
> > shared memory segments. The problem is that a shared memory segment could be
> > mapped to different addresses in different processes. Thus, you want to hold
> > onto offsets into a shared memory segment. apr_rmm.h helps with managing
> > these subblocks and working with offsets rather than direct pointers. Note
> > that apr_rmm also handles locking so that you can have multiple processes
> > allocating (simultaneously) from a shared mem segment.
> >
> > You can then layer additional Python facilities on top of this substrate.
> >
> > Cheers,
> > -g
> >
> > On Wed, Jun 25, 2003 at 07:32:14AM -0700, Jonathan Gardner wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > I am thinking of implementing a shared memory module that does more than just
> > > "read" and "write". Instead, I want something more transparent to the
> > > developer, so that you say "Go get this dict at this shared memory key" or
> > > "Create a new dict in shared memory at this key", and have the entire dict
> > > available like a namespace.
> > >
> > > Obvious applications would include the ever-present problem of shared data
> > > between apache mod_python processes. This would solve it once and for all.
> > > Perhaps one day we could get it working with RPC and have processes on
> > > seperate machines sharing data as well.
> > >
> > > The scope of this project would be implimenting new types that derive from
all
> > > of the basic types so that:
> > >
> > > 1) When accessing the data in the type, it would gain a lock on the shared
> > > memory segment that stores its data using a semaphore, read the data and
> > > unserialize it, then return that data.
> > > 2) When storing the data in that type, it would gain a lock on the shared
> > > memory segment, then store the data over the existing data.
> > >
> > > I am thinking there are really two ways to go about this.
> > > 1) Implementing it all in Python. The idea here would be that we would
> > > serialize and unserialize the data using cPickle.
> > > 2) Implementing it all in C. The idea here is that we would reuse the same
> > > data structures that the basic data types use. We would store and read the
> > > binary data behind the basic data types so that they can be as easily stored
> > > in shared memory as they are stored in real memory.
> > >
> > > Does anyone know if something like this is implemented or in the process of
> > > being implemented? Does anyone want to join me in this endeavor?
> > >
> > > - --
> > > Jonathan Gardner <jgardner@jonathangardner.net>
> > > (was jgardn@alumni.washington.edu)
> > > Live Free, Use Linux!
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG v1.2.1 (GNU/Linux)
> > >
> > > iD8DBQE++bJuWgwF3QvpWNwRAq0pAJ4ipACeWt63fSbtheNmxQfc217sQwCgj4D7
> > > /ID5Uq/oK8hAYQfmvvIGliE=
> > > =4zo4
> > > -----END PGP SIGNATURE-----
> > >
> > > _______________________________________________
> > > Mod_python mailing list
> > > Mod_python@modpython.org
> > > http://mailman.modpython.org/mailman/listinfo/mod_python
> >
> > --
> > Greg Stein, http://www.lyra.org/
> >
>

Mime
View raw message