flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12294) Kafka connector, work with grouping partitions
Date Mon, 29 Apr 2019 12:09:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829187#comment-16829187

Sergey commented on FLINK-12294:

[~gjy], ok and thnx for link with current discussion

> Kafka connector, work with grouping partitions
> ----------------------------------------------
>                 Key: FLINK-12294
>                 URL: https://issues.apache.org/jira/browse/FLINK-12294
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Kafka, Runtime / Coordination
>            Reporter: Sergey
>            Priority: Major
>              Labels: performance
> Additional flag (with default false value) controlling whether topic partitions already
grouped by the key. Exclude unnecessary shuffle/resorting operation when this parameter set
to true. As an example, say we have client's payment transaction in a kafka topic. We grouping
by clientId (transaction with the same clientId goes to one kafka topic partition) and the
task is to find max transaction per client in sliding windows. In terms of map\reduce there
is no needs to shuffle data between all topic consumers, may be it`s worth to do within each
consumer to gain some speedup due to increasing number of executors within each partition
data. With N messages (in partition) instead of N*ln(N) (current realization with shuffle/resorting)
it will be just N operations. For windows with thousands events - the tenfold gain of execution

This message was sent by Atlassian JIRA

View raw message