flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zentol <...@git.apache.org>
Subject [GitHub] flink pull request #4080: [FLINK-6798][docs] update old network buffer notic...
Date Wed, 07 Jun 2017 14:37:26 GMT
Github user zentol commented on a diff in the pull request:

    --- Diff: docs/monitoring/large_state_tuning.md ---
    @@ -94,21 +94,11 @@ When a savepoint is manually triggered, it may be in process concurrently
with a
     ## Tuning Network Buffers
    -The number of network buffers is a parameter that can currently have an effect on checkpointing
at large scale.
    -The Flink community is working on eliminating that parameter in the next versions of
    -The number of network buffers defines how much data a TaskManager can hold in-flight
before back-pressure kicks in.
    -A very high number of network buffers means that a lot of data may be in the stream network
channels when a checkpoint
    -is started. Because the checkpoint barriers travel with that data (see [description of
how checkpointing works](../internals/stream_checkpointing.html)),
    -a lot of in-flight data means that the barriers have to wait for that data to be transported/processed
before arriving
    -at the target operator.
    -Having a lot of data in-flight also does not speed up the data processing as a whole.
It only means that data is picked up faster
    -from the data source (log, files, message queue) and buffered longer in Flink. Having
fewer network buffers means that
    -data is picked up from the source more immediately before it is actually being processed,
which is generally desirable.
    -The number of network buffers should hence not be set arbitrarily large, but to a low
multiple (such as 2x) of the
    -minimum number of required buffers.
    +Before Flink 1.3, an increased number of network buffers also caused increased checkpointing
times since
    +keeping more in-flight data also meant that checkpoint barriers got delayed. Since Flink
1.3, the
    --- End diff --
    remove "also" for a better flow.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message