flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leonard Xu (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-16497) Improve default flush strategy for JDBC sink to make it work out-of-box
Date Tue, 02 Jun 2020 10:43:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123622#comment-17123622
] 

Leonard Xu commented on FLINK-16497:
------------------------------------

Thanks [~sunjincheng121] and [~libenchao] 's detailed comments.

I want to supply that the two parameters is just an out-of-box configuration and users may
need to tune according to their specific scenarios.

I'd like to suggest set *interval less than 1s* because we always proposed that FLIINK framework can
offer sub-second latency, most user cases always send smaller data to db and will use other
sink for big scale data. So, we can set *max-rows less than 10/50* as a basic tuning which
can work with *interval* to reduce the DB's pressure efficiently.

WDYT?

> Improve default flush strategy for JDBC sink to make it work out-of-box
> -----------------------------------------------------------------------
>
>                 Key: FLINK-16497
>                 URL: https://issues.apache.org/jira/browse/FLINK-16497
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / JDBC, Table SQL / Ecosystem
>            Reporter: Jark Wu
>            Priority: Critical
>             Fix For: 1.11.0
>
>
> Currently, JDBC sink provides 2 flush options:
> {code}
> 'connector.write.flush.max-rows' = '5000', -- default is 5000
> 'connector.write.flush.interval' = '2s', -- no default value
> {code}
> That means if flush interval is not set, the buffered output rows may not be flushed
to database for a long time. That is a surprising behavior because no results are outputed
by default. 
> So I propose to have a default flush '1s' interval for JDBC sink or default 1 row for
flush size. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message