cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-299) make table directory creation lazy
Date Tue, 11 Aug 2009 19:02:14 GMT


Jonathan Ellis commented on CASSANDRA-299:

what i'm saying is, you should still open() all tables (i don't see the benefit in changing
that, and it would cause some pain i think), but the scanning for data files should not be
per-table but at a higher level to avoid O(N) system calls scanning datafiledirectories, when
it really only needs to be O(1).  (of course the subdirs then need to be scanned per-table).

this involves breaking the current encapsulation but it will make a big difference if we want
to support 10000s of Tables.  (and i think we do.)

> make table directory creation lazy
> ----------------------------------
>                 Key: CASSANDRA-299
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.5
> checking that each subdir for each table is present on startup -- _every_ startup --
could be a real pita.
> i think that to support 100k tables (not impossible, in a hosted-cassandra-as-a-service
scenario) we're going to want to make table dir creation lazy.
> then we would want to make scanning for sstables faster by only doing one listdir call
per datadir, to see which table subdirs are present, and then checking only those for sstable
files.  this would involve some re-org of the onstart code.
> (note that we don't want to prune directories if there are no sstables left in them,
since we'd end up re-creating them at some point anyway; we just want to allow the lack of
a table subdir to imply the same thing as an empty one.)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message