jakarta-jcs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Smuts <aasm...@wisc.edu>
Subject RE: cvscommit:jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/diskAbstractDiskCache.java
Date Thu, 15 Apr 2004 05:24:28 GMT
Yes and no.  The disk cache will optimize the data file on shutdown.  

During optimization, basically, the cache reads the keys.  For each key
it pulls the data from the file and puts it into a new file.  After it
has read allt he keys, it destroys the old file and puts the new one in
its place.  

There is no real time optimization or defragmetation.  This is pretty
tricky and I'm working on a solution now.

I'm going to have a scheduled optimization.  Also, some timeout
background shrinkers that will go through and get rid of expired
elements during the optimization process.  The first phase of this will
take the disk offline temporarily, so you could schedule it for the
middle of the night.  Later, I'll try to do it live.  Either way you run
into the problem of removes and puts that have come in during the
optimization.  Lossing cached entries is not bad, since nothing should
be cashed that can't be recreated (or almost nothing), but serving old
data is bad.  The incoming remove requests are the big problem.

Basically, if you remove an item, the key is removed and the element on
disk will not be found.  It will take up file space until optimization,
which only occurs at shutdown, but it will not be available.

The system is not wildly inefficient.  When you put something into
diskcache it first checks to see if the element already exists.  If so,
if your put is an update, then it checks to see if the serialized value
is less than or equal to the old value.  If so, it reuses the slot.

There is also a purgatory for disk elements.  When an entry is spooled
from memory it goes to purgatory and then a background process puts it
on disk.  If you request it while it is in purgatory the background job
is cancelled and the element goes back to memory.

Aaron

> -----Original Message-----
> From: Corin Moss [mailto:Corin.Moss@tvnz.co.nz]
> Sent: Wednesday, April 14, 2004 9:39 PM
> To: Turbine JCS Developers List
> Subject: RE: cvscommit:jakarta-turbine-
> jcs/src/java/org/apache/jcs/auxiliary/diskAbstractDiskCache.java
> 
> 
> Hi Aaron,
> 
> Regarding this round of fixes, I noticed mention of a fix relating to
> the removal of expired elements.  Does this ensure that things stored
> with the on disk store aren't left there after key deletion?
> 
> 
> I just noticed whilst browsing the code the following:
> 
>     /**
>      * Returns true if the removal was successful; or false if there
is
> nothing
>      * to remove. Current implementation always result in a disk
orphan.
>      *
>      * @return
>      * @param key
>      */
>     public boolean doRemove( Serializable key )
> 
> The bit about the disk orphan (if still true) is in my mind incredibly
> dangerous.  It means that you have no control over the size of your
> cache - which can result in a disk blow-out.
> 
> 
> Am I right in this? I have noticed that my data file is always
growing,
> never shrinking - this would definitely explain this.
> 
> 
> Does anyone have any comments on this? If my assumptions are correct,
> does anyone have a fix in the works? :)
> 
> Thanks,
> 
> Corin
> 
> -----Original Message-----
> From: asmuts@apache.org [mailto:asmuts@apache.org]
> 
> Sent: Thursday, 15 April 2004 5:06 a.m.
> To: jakarta-turbine-jcs-cvs@apache.org
> Subject: cvs
>
commit:jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/diskAbstrac
> tDiskCache.java
> 
> 
> asmuts      2004/04/14 10:06:12
> 
>   Modified:    src/java/org/apache/jcs/utils/locking
ReadWriteLock.java
>                src/java/org/apache/jcs/auxiliary/disk/indexed
>                         IndexedDiskCache.java
>                src/java/org/apache/jcs/auxiliary/disk
>                         AbstractDiskCache.java
>   Log:
>   It should be safe to dispose now.  Kill queue, optimize, then play
> dead.
> 
> 
>   Fixed a problem in the ReadWriteLock class that let the number of
> writelocks get below zero.  This seems to have solved many problems.
> However, I can't find out why there are more done calls than lock
calls.
> Added a break into the wait on the readlock in case noone calls
notify.
> 
> 
>   This may have fixed the mysterious locking problem that I've never
> been able to duplicate.
> 
> 
>   Revision  Changes    Path
>   1.3       +14 -3
>
jakarta-turbine-jcs/src/java/org/apache/jcs/utils/locking/ReadWriteLock.
> java
> 
> 
>   Index: ReadWriteLock.java
>   ===================================================================
>   RCS file:
>
/home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/utils/locking/Read
> WriteLock.java,v
>   retrieving revision 1.2
>   retrieving revision 1.3
>   diff -u -r1.2 -r1.3
>   --- ReadWriteLock.java	22 Aug 2003 11:57:19 -0000	1.2
>   +++ ReadWriteLock.java	14 Apr 2004 17:06:12 -0000	1.3
>   @@ -117,7 +117,7 @@
>            while ( writeLockedThread != null )
>            {
>                log.debug( "readLock wait" );
>   -            wait();
>   +            wait(20);
>                log.debug( "wake up from readLock wait" );
>            }
> 
> 
>   @@ -211,9 +211,20 @@
>                }
>                return;
>            }
>   +
>            if ( Thread.currentThread() == writeLockedThread )
>            {
>   -            outstandingWriteLocks--;
>   +
>   +            //log.info( "outstandingWriteLocks= " +
> outstandingWriteLocks );
>   +            if ( outstandingWriteLocks > 0 )
>   +            {
>   +                outstandingWriteLocks--;
>   +            }
>   +            else
>   +            {
>   +                log.warn( "extra lock release, writelocks are " +
> outstandingWriteLocks + "and done was called" );
>   +            }
>   +
>                if ( outstandingWriteLocks > 0 )
>                {
>                    log.debug( "writeLock released for a nested
writeLock
> request." );
>   @@ -243,7 +254,7 @@
>                    if ( waitingForReadLock > 0 )
>                    {
>                        log.debug( "writeLock released, notified
waiting
> readers" );
>   -
> 
>   +
>                        notifyAll();
>                    }
>                    else
> 
> 
> 
> 
> 
> 
>   1.9       +18 -2
>
jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/indexed/Index
> edDiskCache.java
> 
> 
>   Index: IndexedDiskCache.java
>   ===================================================================
>   RCS file:
>
/home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/ind
> exed/IndexedDiskCache.java,v
>   retrieving revision 1.8
>   retrieving revision 1.9
>   diff -u -r1.8 -r1.9
>   --- IndexedDiskCache.java	14 Apr 2004 06:24:18 -0000	1.8
>   +++ IndexedDiskCache.java	14 Apr 2004 17:06:12 -0000	1.9
>   @@ -300,6 +300,7 @@
>                {
>                    storageLock.done();
>                }
>   +
>                if ( log.isDebugEnabled() )
>                {
>                    log.debug( "Put to file: " + fileName +
>   @@ -469,8 +470,15 @@
>                }
>                else
>                {
>   +
>   +                if ( log.isDebugEnabled() )
>   +                {
>   +                    log.debug( "Disk removal: Removed from key
hash,
> key " + key );
>   +                }
>   +
>                    // remove single item.
>                    return keyHash.remove( key ) != null;
>   +
>                }
>            }
>            catch ( Exception e )
>   @@ -588,7 +596,14 @@
>            finally
>            {
>                alive = false;
>   -            storageLock.done();
>   +
>   +            try
>   +            {
>   +              storageLock.done();
>   +            } catch ( Exception e )
>   +            {
>   +                log.error( "Failure releasing lock on shutdown " +
e
> );
>   +            }
>            }
>        }
> 
> 
>   @@ -673,6 +688,7 @@
>                        log.debug( fileName + " -- newData.length() = "
+
>                            newData.length() );
>                    }
>   +
>                    newData.renameTo( newFileName );
>                }
>                keyHash = keyHashTemp;
> 
> 
> 
> 
> 
> 
>   1.19      +4 -6
>
jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskC
> ache.java
> 
> 
>   Index: AbstractDiskCache.java
>   ===================================================================
>   RCS file:
>
/home/cvs/jakarta-turbine-jcs/src/java/org/apache/jcs/auxiliary/disk/Abs
> tractDiskCache.java,v
>   retrieving revision 1.18
>   retrieving revision 1.19
>   diff -u -r1.18 -r1.19
>   --- AbstractDiskCache.java	14 Apr 2004 06:24:18 -0000	1.18
>   +++ AbstractDiskCache.java	14 Apr 2004 17:06:12 -0000	1.19
>   @@ -300,13 +300,11 @@
>        public final void dispose()
>        {
> 
> 
>   -        // Invoke any implementation specific disposal code
>   -
>   -        doDispose();
>   -
>            // FIXME: May lose the end of the queue, need to be more
> graceful
>   -
>            cacheEventQueue.destroy();
>   +
>   +        // Invoke any implementation specific disposal code
>   +        doDispose();
> 
> 
>            alive = false;
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: turbine-jcs-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-dev-help@jakarta.apache.org
> 
> 
> ================================================================
> CAUTION: This e-mail and any attachment(s) contains information
> that is intended to be read only by the named recipient(s). It
> may contain information that is confidential, proprietary or the
> subject of legal privilege. This information is not to be used by
> any other person and/or organisation. If you are not the intended
> recipient, please advise us immediately and delete this e-mail
> from your system. Do not use any information contained in it.
> 
> ================================================================
> For more information on the Television New Zealand Group, visit
> us online at http://www.tvnz.co.nz
> ================================================================
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: turbine-jcs-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-dev-help@jakarta.apache.org


Mime
View raw message