flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From uce <...@git.apache.org>
Subject [GitHub] incubator-flink pull request: [FLINK-1157] Document TaskManager sl...
Date Mon, 01 Dec 2014 18:05:44 GMT
Github user uce commented on a diff in the pull request:

    --- Diff: docs/config.md ---
    @@ -266,3 +272,79 @@ So if `yarn.am.rpc.port` is configured to `10245` and the session's
     - `yarn.am.rpc.port`: The port that is being opened by the Application Master (AM) to

     let the YARN client connect for an RPC serice. (DEFAULT: Port 10245)
    +## Background
    +### Configuring the Network Buffers
    +Network buffers are a critical resource for the communication layers. They are
    +used to buffer records before transmission over a network, and to buffer
    +incoming data before dissecting it into records and handing them to the
    +application. A sufficient number of network buffers are critical to achieve a
    +good throughput.
    +In general, configure the task manager to have so many buffers that each logical
    +network connection on you expect to be open at the same time has a dedicated
    +buffer. A logical network connection exists for each point-to-point exchange of
    +data over the network, which typically happens at repartitioning- or
    +broadcasting steps. In those, each parallel task inside the TaskManager has to
    +be able to talk to all other parallel tasks. Hence, the required number of
    +buffers on a task manager is *total-degree-of-parallelism* (number of targets)
    +\* *intra-node-parallelism* (number of sources in one task manager) \* *n*.
    +Here, *n* is a constant that defines how many repartitioning-/broadcasting steps
    +you expect to be active at the same time.
    +Since the *intra-node-parallelism* is typically the number of cores, and more
    +than 4 repartitioning or broadcasting channels are rarely active in parallel, it
    +frequently boils down to *\#cores\^2\^* \* *\#machines* \* 4. To support for
    +example a cluster of 20 8-core machines, you should use roughly 5000 network
    +buffers for optimal throughput.
    +Each network buffer is by default 64 KiBytes large. In the above example, the
    +system would allocate roughly 300 MiBytes for network buffers.
    +The number and size of network buffers can be configured with the following
    +- `taskmanager.network.numberOfBuffers`, and
    +- `taskmanager.network.bufferSizeInBytes`.
    +### Configuring Temporary I/O Directories
    +Although Flink aims to process as much data in main memory as possible,
    +it is not uncommon that  more data needs to be processed than memory is
    --- End diff --
    double white space

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message