jakarta-jcs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Smuts <asm...@yahoo.com>
Subject RE: High Data Volume Issues
Date Wed, 19 Jul 2006 19:48:54 GMT


--- "Schwarz, Peter" <peter.schwarz@objectfx.com>
wrote:

> This depends on how your test is written.  One thing
> that we noticed was
> that the cache loses writes of data (through the
> purgatory), if you don't
> wait long enough.  We had, for 2 million items, each
> 16k, a cache size of
> 33GB, which was inline with the computed size for
> the cache.  
> 

There is an optimization, where items are removed from
the queue if they are plucked out of purgatory.  This
would cause problems for memory size = 0
configurations, since the item will be lost.  I'm
working on a fix since I run in a 0 memory
configuration on my remote server cluster.

> I'm not sure what MySQL would be doing to keep that
> size down. 
> 

MySQL uses more space if anything.  I'm just using it
so I can have a background cleanup process.  I had to
build in processes to optimize myisam on a schedule.  

> As for the memory limitations, the key set is
> actually not too bad, since
> we're using pretty small keys (Integers for the
> tests).  With a 512M vm
> size, this wasn't an issue.

Very interesting.  I'm mainly running multiple small <
128MB max apps on a box.  . . .

> 
> > The optimization routine is fairly crude.   
> 
> Using a database isn't an option for us (I know, it
> sounds strange, but hey,
> sometimes the powers that be make certain calls...),
> so we're planning on
> making changes to the defrag/recycle portions of the
> IndexDiskCache.  We
> would like to submit the changes for contribution. 
> What would the process
> be for that?

I'd love to see improvements in this part of the
indexed disk cache.  Please send patches along with
unit test so I can evaluate them.  What exactly are
you doing?  Do you want to optimize in place using a
larger recycle bin?  

If the changes are major, we can consider adding
another disk cache implemention in addition to the
ones that we currently support.

Cheers,

Aaron

> 
> Cheers, 
> 
> Peter
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> jcs-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> jcs-dev-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-dev-help@jakarta.apache.org


Mime
View raw message