cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Yeschenko (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
Date Tue, 03 Jan 2017 17:54:58 GMT


Aleksey Yeschenko updated CASSANDRA-12171:
    Assignee:     (was: Aleksey Yeschenko)

> counter mismatch during rolling upgrade from 2.2 to 3.0
> -------------------------------------------------------
>                 Key: CASSANDRA-12171
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Russ Hatch
> This may occur on other versions, but 3.0 is where I observed it recently.
> N=RF=3, counter writes at quorum, reads at quorum.
> This is being seen on some upgrade tests I'm currently repairing here:
(this branch is to resolve an issue where counters were not being properly tested during rolling
upgrade tests).
> The test runs a continuous counter incrementing process, as well as a continuous counter
checking process. Once a counter value has been verified, the test code makes it eligible
to be incremented again.
> The test is encountering the problem when trying to check an expected counter value and
not matching expectations, for example:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/", line 258, in _bootstrap
>   File "/usr/lib/python2.7/multiprocessing/", line 114, in run
>     self._target(*self._args, **self._kwargs)
>   File "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/",
line 210, in counter_checker
>     tester.assertEqual(expected_count, actual_count)
>   File "/usr/lib/python2.7/unittest/", line 513, in assertEqual
>     assertion_func(first, second, msg=msg)
>   File "/usr/lib/python2.7/unittest/", line 506, in _baseAssertEqual
>     raise self.failureException(msg)
> AssertionError: 1 != 2
> {noformat}
> To check if something else could be going on, I did an experiment where I changed the
test to not upgrade nodes (just drain, stop, start) and the mismatch didn't occur in several
attempts. So it appears something about upgrading is possibly the culprit.
> To run the test and repro locally:
> {noformat}
> grab my dtest branch at
> export UPGRADE_TEST_RUN=true
> nosetests -v upgrade_tests/
> {noformat}

This message was sent by Atlassian JIRA

View raw message