cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksei Zotov (Jira)" <>
Subject [jira] [Commented] (CASSANDRA-16776) modify SecondaryIndexManager#indexPartition() to retrieve only columns for which indexes are actually being built
Date Fri, 09 Jul 2021 21:44:00 GMT


Aleksei Zotov commented on CASSANDRA-16776:

Hi [~maedhroz]

I checked the PR and the code looks good to me. I'm definitely not an expert in that part
of code, but logically your changes are clear and make perfect sense to me.


I'm trying to follow Benjamin's call seeking for reviewers. Definitely a non-committer review
does not let it be merged, but I hope it is still useful.

> modify SecondaryIndexManager#indexPartition() to retrieve only columns for which indexes
are actually being built
> -----------------------------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-16776
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Feature/2i Index
>            Reporter: Caleb Rackliffe
>            Assignee: Caleb Rackliffe
>            Priority: Normal
>             Fix For: 4.x
>         Attachments: index1.png, index2.png
> Secondary indexes are (for the moment) built as special compaction tasks via {{SecondaryIndexBuilder}}.
From a profiling perspective, the fun begins in {{SecondaryIndexManager.indexPartition()}}.
The work above it in {{SecondaryIndexBuilder}} is just key iteration.
>  !index1.png! 
> Two basic things happen in {{indexPartition()}}. First, we read a single partition in
its entirety, and then we send individual rows to the {{Indexer}}. When we read these partitions,
we use {{ColumnFilter.all()}}, which ends up materializing full rows, even when we’re indexing
a single column (or at least fewer columns than we need for all the indexes participating
in the build). If we narrowed this to fetch only the necessary columns, we might be able to
create less garbage in {{AbstractBTreePartition#searchIterator()}} when we create a copy of
the underlying full row from disk.
> In some initial testing, I’ve been using a simple schema with fairly narrow rows.
> {noformat}
> CREATE TABLE tlp_stress.allow_filtering (
>     partition_id text,
>     row_id int,
>     payload text,
>     value int,
>     PRIMARY KEY (partition_id, row_id)
> {noformat}
> The price of deserializing these rows is still visible, however, in the results of some
basic sampling profiling.
>  !index2.png! 
> The possible optimization above to avoid unnecessary copying of a row’s columns would
also narrow cell deserialization only to indexed cells, which would probably be very beneficial
for index builds with very wide rows. One minor wrinkle in all of this is that since 3.0,
it has been possible to create indexes one entire rows, rather than single columns, so we’d
have to keep that case in mind.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message