flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1157) Document TaskManager Slots
Date Mon, 01 Dec 2014 18:06:13 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230148#comment-14230148
] 

ASF GitHub Bot commented on FLINK-1157:
---------------------------------------

Github user uce commented on a diff in the pull request:

    https://github.com/apache/incubator-flink/pull/246#discussion_r21106206
  
    --- Diff: docs/config.md ---
    @@ -266,3 +272,79 @@ So if `yarn.am.rpc.port` is configured to `10245` and the session's
application
     
     - `yarn.am.rpc.port`: The port that is being opened by the Application Master (AM) to

     let the YARN client connect for an RPC serice. (DEFAULT: Port 10245)
    +
    +
    +## Background
    +
    +### Configuring the Network Buffers
    +
    +Network buffers are a critical resource for the communication layers. They are
    +used to buffer records before transmission over a network, and to buffer
    +incoming data before dissecting it into records and handing them to the
    +application. A sufficient number of network buffers are critical to achieve a
    +good throughput.
    +
    +In general, configure the task manager to have so many buffers that each logical
    +network connection on you expect to be open at the same time has a dedicated
    +buffer. A logical network connection exists for each point-to-point exchange of
    +data over the network, which typically happens at repartitioning- or
    +broadcasting steps. In those, each parallel task inside the TaskManager has to
    +be able to talk to all other parallel tasks. Hence, the required number of
    +buffers on a task manager is *total-degree-of-parallelism* (number of targets)
    +\* *intra-node-parallelism* (number of sources in one task manager) \* *n*.
    +Here, *n* is a constant that defines how many repartitioning-/broadcasting steps
    +you expect to be active at the same time.
    +
    +Since the *intra-node-parallelism* is typically the number of cores, and more
    +than 4 repartitioning or broadcasting channels are rarely active in parallel, it
    +frequently boils down to *\#cores\^2\^* \* *\#machines* \* 4. To support for
    +example a cluster of 20 8-core machines, you should use roughly 5000 network
    +buffers for optimal throughput.
    +
    +Each network buffer is by default 64 KiBytes large. In the above example, the
    +system would allocate roughly 300 MiBytes for network buffers.
    +
    +The number and size of network buffers can be configured with the following
    +parameters:
    +
    +- `taskmanager.network.numberOfBuffers`, and
    +- `taskmanager.network.bufferSizeInBytes`.
    +
    +### Configuring Temporary I/O Directories
    +
    +Although Flink aims to process as much data in main memory as possible,
    +it is not uncommon that  more data needs to be processed than memory is
    --- End diff --
    
    double white space


> Document TaskManager Slots
> --------------------------
>
>                 Key: FLINK-1157
>                 URL: https://issues.apache.org/jira/browse/FLINK-1157
>             Project: Flink
>          Issue Type: Improvement
>          Components: Documentation
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>
> Slots are not explained in the documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message