cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-13762) Ensure views created during (or just before) range movements are properly built
Date Fri, 05 Jan 2018 12:00:02 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paulo Motta updated CASSANDRA-13762:
------------------------------------
    Status: Patch Available  (was: Open)

The problem in this ticket description is reproduced by this [dtest|https://github.com/apache/cassandra-dtest/compare/master...pauloricardomg:13762#diff-62ba429edee6a4681782f078246c9893R1185].

The fix consists in applying the optimization from CASSANDRA-13065 only if the source node
has built all the views from the streamed base table. Otherwise, the mutations are sent through
the write path as it was done before.

In order to check if a source node has built all views we need to query the {{system_distrbuted.view_build_status}}
table. Querying this table on a bootstrapping node was [previously throwing|https://github.com/pauloricardomg/cassandra/blob/c47cfb9f3299274f97e47610eb6329390dcacdc0/src/java/org/apache/cassandra/service/StorageProxy.java#L1619]
{{IsBootstrappingException}}, but I didn't find any good reason for this limitation, so I
lifted it [here|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-13762#diff-71f06c193f5b5e270cf8ac695164f43aR1580].

One problem that arises when sending mutations through the write path during bootstrap is
that the views are not propagated to replicas, but [written to the batchlog|https://github.com/apache/cassandra/blob/e18a49a2399a3fe667c3c08d7350b7528614f0a6/src/java/org/apache/cassandra/service/StorageProxy.java#L831]
so the views may not be fully available when the node finishes bootstrapping, even though
they are [marked as built|https://github.com/apache/cassandra/blob/e18a49a2399a3fe667c3c08d7350b7528614f0a6/src/java/org/apache/cassandra/service/StorageService.java#L1537].
This is the same problem described on CASSANDRA-13629. 

The fix to the problem above is to [wait for batchlog replay|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-13762#diff-b76a607445d53f18a98c9df14323c7ddR1543]
before finishing bootstrap if views are present (we should probably reopen CASSANDRA-13629
and backport this fix btw). One downside of this approach is that we need to [wait for the
batchlog timeout|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-13762#diff-b76a607445d53f18a98c9df14323c7ddR1541]
to ensure all batchlogs are replayed, what should increase the bootstrap in tens of seconds,
but should be acceptable. If necessary we can try to raise this limitation in another ticket.

I also added a dtest to check that view rebuild is skipped during streaming if the source
node has built the views ([here|https://github.com/apache/cassandra-dtest/compare/master...pauloricardomg:13762#diff-62ba429edee6a4681782f078246c9893R1189]).

Patch and tests below (CI looks good):

||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-13762]|[branch|https://github.com/apache/cassandra-dtest/compare/master...pauloricardomg:13762]|
|[testall|https://issues.apache.org/jira/secure/attachment/12904779/trunk-13762-testall.png]|
|[dtest|https://issues.apache.org/jira/secure/attachment/12904778/trunk-13762-dtest.png]|

> Ensure views created during (or just before) range movements are properly built
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13762
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13762
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Materialized Views
>            Reporter: Paulo Motta
>            Assignee: Paulo Motta
>            Priority: Minor
>              Labels: materializedviews
>         Attachments: trunk-13762-dtest.png, trunk-13762-testall.png
>
>
> CASSANDRA-13065 assumes the source node has its views built to skip running base mutations
through the write path during range movements.
> It is possible that the source node has not finished building the view, or that a new
view is created during a range movement, in which case the view may be wrongly marked as built
on the destination node.
> The former problem was introduced by #13065, but even before that a view created during
a range movement may not be correctly built on the destination node because the view builder
will be triggered before it has finished streaming the source data, wrongly marking the view
as built on that node.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message