cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Meredith (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-14790) LongBufferPoolTest burn test fails assertion
Date Wed, 17 Oct 2018 15:23:00 GMT


Jon Meredith commented on CASSANDRA-14790:

[~benedict] I think we're reading it the same way - my argument was that it can cause buffers
to be allocated from the heap when MEMORY_USAGE_THRESHOLD has not been exceeded yet. I'd describe
it as a benign race rather than beneficial. The calling thread has to pay the price of allocating
chunks that other threads stole and then an extra allocation which could possibly result in
a blocking system call to get more memory. Instead allocateMoreChunks could return one of
the chunks to it's caller and add one less chunks to the queue.

I'm not even sure it's worth changing anything, but [~djoshi3] wanted to see what you thought
about it.

 Here's the example I wrote up before I read your comment more carefully.

Start with no allocations from any of the thread local or buffer pools yet.

 MEMORY_USAGE_THRESHOLD = 16384 KiB (for the unit test)

1) T1 calls BufferPool.get(1) and ends up in GlobalPool:get. chunks.poll returns null so it
calls allocateMoreChunks which allocates a macro chunk, divides it up into 16 (1024KiB / 64KiB)
Chunks that are added to BufferPool.GlobalPool.chunks.

2) Between the adding the last chunk and the 'one last attempt' to pull it in Chunk.get, 16
other calls to GlobalPool::get take place on other threads, emptying GlobalPool.chunks

3) T1 returns from allocateMoreChunks, back in Chunk::get chunks.poll() returns null and which
gets passed up the call chain with the null causing a call to BufferPool.allocate which allocates
memory outside of the pool, despite the current pool memory usage being at ~1MiB, which is
less than the usage threshold and should have been satisfied by the pool.

As I said, I don't think it's really a big deal as memory allocated outside the pool should
be freed/garbage collected just fine and the buffer pool is just an optimization.

It's also possible for T1, T2 to both arrive in allocateMoreBuffers with BufferPool.GlobalPool.chunk
 empty and cause harmless allocation of extra buffers, but it looks like it uses atomics
 to make sure the MEMORY_USAGE_THRESHOLD invariant isn't exceeded.

> LongBufferPoolTest burn test fails assertion
> --------------------------------------------
>                 Key: CASSANDRA-14790
>                 URL:
>             Project: Cassandra
>          Issue Type: Test
>          Components: Testing
>         Environment: Run under macOS 10.13.6, with patch (attached, but also
>            Reporter: Jon Meredith
>            Assignee: Jon Meredith
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: 0001-Add-burn-testsome-target-to-build.xml.patch, 0002-Initialize-before-running-LongBufferPoolTest.patch
>          Time Spent: 2h
>  Remaining Estimate: 0h
> The LongBufferPoolTest from the burn tests fails with an assertion error.  I added a
build target to run individual burn tests, and \{jasobrown} gave a fix for the uninitialized
test setup (attached), however the test now fails on an assertion about recycling buffers.
> To reproduce (with patch applied)
> {{ant burn-testsome
> Output
> {{    [junit] Testcase: testAllocate(org.apache.cassandra.utils.memory.LongBufferPoolTest):
> {{    [junit] null}}
> {{    [junit] junit.framework.AssertionFailedError}}
> {{    [junit] at org.apache.cassandra.utils.memory.BufferPool$Debug.check(}}
> {{    [junit] at org.apache.cassandra.utils.memory.BufferPool.assertAllRecycled(}}
> {{    [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest.testAllocate(}}
> {{    [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest.testAllocate(}}
> All major branches from 3.0 and later have issues, however the trunk branch also warns about
references not being released before the reference is garbage collected.
> {{[junit] ERROR [Reference-Reaper:1] 2018-09-25 13:59:54,089 - LEAK DETECTED:
a reference (org.apache.cassandra.utils.concurrent.Ref$State@7f58d19a) to @623704362 was not
released before the reference was garbage collected}}
> {{ [junit] ERROR [Reference-Reaper:1] 2018-09-25 13:59:54,089 - Allocate
trace org.apache.cassandra.utils.concurrent.Ref$State@7f58d19a:}}
> {{ [junit] Thread[pool-2-thread-24,5,main]}}
> {{ [junit] at java.lang.Thread.getStackTrace(}}
> {{ [junit] at org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(}}
> {{ [junit] at org.apache.cassandra.utils.concurrent.Ref$State.<init>(}}
> {{ [junit] at org.apache.cassandra.utils.concurrent.Ref.<init>(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool$Chunk.setAttachment(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool$Chunk.get(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool$Chunk.get(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(}}
> {{ [junit] at org.apache.cassandra.utils.memory.BufferPool.get(}}
> {{ [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest$3.allocate(}}
> {{ [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest$3.testOne(}}
> {{ [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest$}}
> {{ [junit] at org.apache.cassandra.utils.memory.LongBufferPoolTest$}}
> {{ [junit] at}}
> {{ [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(}}
> {{ [junit] at java.util.concurrent.ThreadPoolExecutor$}}
> {{ [junit] at}}
> Perhaps the environment is not being set up correctly for the tests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message