quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gregory (Grisha) Trubetskoy" <gri...@apache.org>
Subject Re: [mod_python] Using shared memory to do global persistence
Date Tue, 08 Jul 2003 20:08:59 GMT

Thanks for the tip on rmm. I've been looking at the code and mod_ssl seems
to use it extensively, in fact mod_ssl has a usable implementation of a
shared memory hash table used for session tracking, even with timeouts..

I am not very clear why mod_ssl uses rmm, other than as a way to allocate
and free blocks of memory within a large shared memory segment.

E.g. its hash table (which, btw is not an APR hash table), which is able
to use pluggable malloc funcs (those in turn use rmm), still stores real
pointers, and to free memory mod_ssl does stuff like [slightly
simplified]:

    apr_rmm_off_t off = apr_rmm_offset_get(RMM, ptr);
    apr_rmm_free(RMM, off);

Either I'm missing something, or converting a real pointer (possibly
originally created in a different process) to an offset via
apr_rmm_offset_get() is not a safe thing to do if access to rmm was
obtained via apr_rmm_attach(), as opposed to apr_rmm_init(). (Mod_ssl does
not need to use apr_rmm_attach(), which is why it works I think)

I suppose for our purposes, the same strategy (doing rmm_init at module
initialization/pre-fork time) might work, though it would be nice if we
could use an APR hash table that stored rmm offsets (I haven't looked at
that yet, but the fact that mod_ssl uses a home-grown table implementation
suggests that it may not be possible) this way shared memory could be
created as-needed from within child processes.

Grisha

On Thu, 26 Jun 2003, Greg Stein wrote:

> APR has facilities to do shared memory in a portable fashion; APR is part of
> Apache 2.0, so there isn't much reason to go grab any other library.
>
> You can also use APRUTIL's apr_rmm.h to manage sub-allocations within the
> shared memory segments. The problem is that a shared memory segment could be
> mapped to different addresses in different processes. Thus, you want to hold
> onto offsets into a shared memory segment. apr_rmm.h helps with managing
> these subblocks and working with offsets rather than direct pointers. Note
> that apr_rmm also handles locking so that you can have multiple processes
> allocating (simultaneously) from a shared mem segment.
>
> You can then layer additional Python facilities on top of this substrate.
>
> Cheers,
> -g
>
> On Wed, Jun 25, 2003 at 07:32:14AM -0700, Jonathan Gardner wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I am thinking of implementing a shared memory module that does more than just
> > "read" and "write". Instead, I want something more transparent to the
> > developer, so that you say "Go get this dict at this shared memory key" or
> > "Create a new dict in shared memory at this key", and have the entire dict
> > available like a namespace.
> >
> > Obvious applications would include the ever-present problem of shared data
> > between apache mod_python processes. This would solve it once and for all.
> > Perhaps one day we could get it working with RPC and have processes on
> > seperate machines sharing data as well.
> >
> > The scope of this project would be implimenting new types that derive from all
> > of the basic types so that:
> >
> > 1) When accessing the data in the type, it would gain a lock on the shared
> > memory segment that stores its data using a semaphore, read the data and
> > unserialize it, then return that data.
> > 2) When storing the data in that type, it would gain a lock on the shared
> > memory segment, then store the data over the existing data.
> >
> > I am thinking there are really two ways to go about this.
> > 1) Implementing it all in Python. The idea here would be that we would
> > serialize and unserialize the data using cPickle.
> > 2) Implementing it all in C. The idea here is that we would reuse the same
> > data structures that the basic data types use. We would store and read the
> > binary data behind the basic data types so that they can be as easily stored
> > in shared memory as they are stored in real memory.
> >
> > Does anyone know if something like this is implemented or in the process of
> > being implemented? Does anyone want to join me in this endeavor?
> >
> > - --
> > Jonathan Gardner <jgardner@jonathangardner.net>
> > (was jgardn@alumni.washington.edu)
> > Live Free, Use Linux!
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.2.1 (GNU/Linux)
> >
> > iD8DBQE++bJuWgwF3QvpWNwRAq0pAJ4ipACeWt63fSbtheNmxQfc217sQwCgj4D7
> > /ID5Uq/oK8hAYQfmvvIGliE=
> > =4zo4
> > -----END PGP SIGNATURE-----
> >
> > _______________________________________________
> > Mod_python mailing list
> > Mod_python@modpython.org
> > http://mailman.modpython.org/mailman/listinfo/mod_python
>
> --
> Greg Stein, http://www.lyra.org/
>

Mime
View raw message