cassandra-pr mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jonmeredith <...@git.apache.org>
Subject [GitHub] cassandra pull request #279: CASSANDRA-14790 Fix flaky LongBufferPoolTest
Date Sun, 07 Oct 2018 20:21:04 GMT
GitHub user jonmeredith opened a pull request:

    https://github.com/apache/cassandra/pull/279

    CASSANDRA-14790 Fix flaky LongBufferPoolTest

    The LongBufferPoolTest previously required significantly more heap memory on machines
with higher core count.
    
    This PR
    - adds commands to the build system to allow running individual burn tests (in and outside
junit)
    - fixes some race conditions that occur when the test is running under heavy memory pressure
    - changes the calculation for how much memory the ring-of-threads should allocate to be
roughly double the pool size under test.  It now completes with much less memory and should
run fine on a builder.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jonmeredith/cassandra CASSANDRA-14790-3.0

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/cassandra/pull/279.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #279
    
----
commit 0f55a39849ff4fa3b0f2b4f0d61678e08fa03a46
Author: Jon Meredith <jmeredithco@...>
Date:   2018-10-04T23:08:52Z

    Add build targets for running burn tests.

commit 00abab9ccbfd0d6a154c31a69c0780f142df5121
Author: Jon Meredith <jmeredithco@...>
Date:   2018-10-04T23:10:02Z

    Harden LongBufferPoolTest against java.lang.OutOfMemoryError
    
    Catch all possible errors on the worker threads, otherwise if threads
    exit due to java.lang.OutOfMemoryError, the test either reports
    a lack of progress or the chunk recycling check fails.
    
    Add an explicit exit to the main on errors as the REQUEST-SCHEDULER
    thread is not marked as a daemon and prevents the test from exiting.
    
    Added some extra logging to clear up how large the buffer pool
    size is under test, as loading the default config prints
    `Global buffer pool is enabled, when pool is exahusted (max is 512 mb)
    it will allocate on heap`
    and that is not what gets tested.

commit db755d706faa363a4973b371d68994db65b8be83
Author: Jon Meredith <jmeredithco@...>
Date:   2018-10-07T19:47:49Z

    Harden LongBufferPoolTest against flaky Chunk recycle assertion
    
    The test as written does not guarantee that all Chunks
    will be recycled, just that it is likely that once during
    the 10s check cycle, worker threads sets the target memory size
    to zero (a one-in-16k chance).
    
    Instead, make the thread free at least once per cycle and
    only do the recycle test when all threads are
    known to have released all memory as they might exit.
    
    Also, check the return code from the burn producer and consumer
    threads in case they throw an assertion error. The current burn
    producer/consumer threads currently double-free each buffer, which
    occasionally triggers a double-release assertion check.

commit b2b509a4ed35a27cc7a90f47cc8f47d2683c9ea1
Author: Jon Meredith <jmeredithco@...>
Date:   2018-10-04T23:29:54Z

    Change target allocation size for LongBufferPoolSize.
    
    The prior method for calculating the amount of memory to try
    and allocate for the tests increased when it ran with more threads.
    The pool under test is a fixed size pool so the effect was to add
    pressure on heap allocations instead.  On a 6-core i7 the test failed
    with -Xmx500m.
    
    This change restricts the total allocation size to a little over
    double the pool size regardless of the number of threads present.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Mime
View raw message