flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanping Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2343) Change default garbage collector in streaming environments
Date Mon, 13 Jul 2015 18:09:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625064#comment-14625064

Yanping Wang commented on FLINK-2343:

Yes, G1 for jdk8+ is a good choice. If we hit functional or performance issues, we still have
chance to fix them in G1 for Flink.

Regarding CMS, minor GC from CMS does not do heap compaction at all, it relies on major GC
(single threaded full GC) to collect and compact entire heap. So CMS will likely hit the long
GC pauses when heap size goes above 16GB. 
I tested 160GB heap with CMS with the best tuning I could do on the same spark graph computing
test, CMS completed the task in ~350 seconds, while jdk8u60 with the optimizations, completed
the task in 140-145 seconds. 

I'd love to see how G1 and CMS run on Flink tests. Let me know if there is any issues you
run into.

> Change default garbage collector in streaming environments
> ----------------------------------------------------------
>                 Key: FLINK-2343
>                 URL: https://issues.apache.org/jira/browse/FLINK-2343
>             Project: Flink
>          Issue Type: Improvement
>          Components: Start-Stop Scripts
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.10
> When starting Flink, we don't pass any particular GC related JVM flags to the system.
That means, it uses the default garbage collectors, which are the bulk parallel GCs for both
old gen and new gen.
> For streaming applications, this results in vastly fluctuating latencies. Latencies are
much more constant with either the {{CMS}} or {{G1}} GC.
> I propose to make the CMS the default GC for streaming setups.
> G1 may become the GC of choice in the future, but fro various articles I found, it is
still somewhat in "beta" status (see for example here: http://jaxenter.com/kirk-pepperdine-on-the-g1-for-java-9-118190.html

This message was sent by Atlassian JIRA

View raw message