cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14239) OutOfMemoryError when bootstrapping with less than 100GB RAM
Date Wed, 11 Apr 2018 15:09:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434047#comment-16434047
] 

Paulo Motta commented on CASSANDRA-14239:
-----------------------------------------

bq. Number of pending MemtableFlushWriter jobs is slooooowly decreasing,  I'll try to wait
till it decrease to zero, maybe this will unblock mutations.

Perhaps you could try setting the system property {{-Dcassandra.repair.mutation_repair_rows_per_batch=1000}}
(from the default 100) and see if this will make the pending queue decrease faster while keeping
the GC sane.

bq. Ok. Now it stuck on 5134 pending MemtableFlushWriter jobs. Number is not decreasing anymore.
 

Can you attach a thread dump? You can generate it via jstack <pid>

> OutOfMemoryError when bootstrapping with less than 100GB RAM
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-14239
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Details of the bootstrapping Node
>  * ProLiant BL460c G7
>  * 56GB RAM
>  * 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and saved_caches)
>  * CentOS 7.4 on SD-Card
>  * /tmp and /var/log on tmpfs
>  * Oracle JDK 1.8.0_151
>  * Cassandra 3.11.1
> Cluster
>  * 10 existing Nodes (Up and Normal)
>            Reporter: J├╝rgen Albersdorfer
>            Priority: Major
>              Labels: materializedviews
>         Attachments: Objects-by-class.csv, Objects-with-biggest-retained-size.csv, Selection_420.png,
Selection_421.png, cassandra-env.sh, cassandra.yaml, dstat.png, gc.log.0.201804111524.zip,
gc.log.0.current.zip, gc.log.201804111141.zip, jvm.options, jvm_opts.txt, stack-traces.txt
>
>
> Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our 10 Node
C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen
which gets not significantly freed up any more.
> I know that JVM collects on Old Gen only when really needed. I can see collections, but
there is always a remainder which seems to grow forever without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra RAM I have given
it for bootstrapping without any further effect.
> It feels like Cassandra will not forget about every single byte streamed over the Network
over time during bootstrapping, - which would be a memory leak and a major problem, too.
> I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB assigned JVM
Heap). YourKit Profiler shows huge amount of Memory allocated for org.apache.cassandra.db.Memtable
(22 GB) org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11 GB)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message