flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanping Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2343) Change default garbage collector in streaming environments
Date Sun, 12 Jul 2015 16:32:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623879#comment-14623879
] 

Yanping Wang commented on FLINK-2343:
-------------------------------------

Hi, Stephan

There were some bug fixes and optimizations back ported to jdk7 update 60 and later but not
all, and not up to date. The most recent or last jdk7, which is update 79, http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
  that probably is the best G1 in jdk7 you can get. As jdk7 is no longer to be updated, update
79 was marked the end of jdk7.

The most helpful optimizations and fixes were in upcoming jdk8 update 60 (scheduled to be
default java download in sep/oct 2015) , and jdk9 a year later.

The spark graph computing workload performance I measured 2X performance gain on 160GB heap
was compare jdk7u79 vs. jdk8u60. Also there is an article on tuning G1 for Spark was based
on jdk8u40 from my fellow engineers working on Spark (my name was listed at ACK section :-)
https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html
 

One good point by using G1 is, if we find problems, we can ask Oracle Java developers to fix
and to optimize. For CMS, there is no development efforts there, as it will be replaced by
G1. 

By the way, in the experiment I did and also mentioned by the paper above, CMS was bad because
the Spark workload was a throughput workload with low latency requirement. Throughput was
run down so much when CMS running into full GC to compact a large heap. 

If you run G1 on Flink and see funning things, let me know I think I can try making it better.

Thanks
-yanping



> Change default garbage collector in streaming environments
> ----------------------------------------------------------
>
>                 Key: FLINK-2343
>                 URL: https://issues.apache.org/jira/browse/FLINK-2343
>             Project: Flink
>          Issue Type: Improvement
>          Components: Start-Stop Scripts
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.10
>
>
> When starting Flink, we don't pass any particular GC related JVM flags to the system.
That means, it uses the default garbage collectors, which are the bulk parallel GCs for both
old gen and new gen.
> For streaming applications, this results in vastly fluctuating latencies. Latencies are
much more constant with either the {{CMS}} or {{G1}} GC.
> I propose to make the CMS the default GC for streaming setups.
> G1 may become the GC of choice in the future, but fro various articles I found, it is
still somewhat in "beta" status (see for example here: http://jaxenter.com/kirk-pepperdine-on-the-g1-for-java-9-118190.html
)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message