flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (Jira)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode
Date Tue, 01 Oct 2019 16:41:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Piotr Nowojski updated FLINK-10995:
-----------------------------------
        Fix Version/s: 1.10.0
    Affects Version/s: 1.9.0

> Copy intermediate serialization results only once for broadcast mode
> --------------------------------------------------------------------
>
>                 Key: FLINK-10995
>                 URL: https://issues.apache.org/jira/browse/FLINK-10995
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Task
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 1.10.0
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> The emitted records from operator would be firstly serialized into intermediate bytes
array in {{RecordSerializer}}, then copy the intermediate results into target buffers for
different sub partitions.  For broadcast mode, the same intermediate results would be copied
as many times as the number of sub partitions, and this would affect the performance seriously
in large scale jobs.
> We can copy to only one target buffer which would be shared by all the sub partitions
to reduce the overheads. For emitting latency marker in broadcast mode, we should flush the
previous shared target buffers first, and then request a new buffer for the target sub partition
to send latency marker.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message