flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jark Wu (Jira)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-16497) Improve default flush strategy for JDBC sink to make it work out-of-box
Date Mon, 08 Jun 2020 10:52:01 GMT

     [ https://issues.apache.org/jira/browse/FLINK-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jark Wu updated FLINK-16497:
----------------------------
    Description: 
Currently, old JDBC sink provides 2 flush options:

{code}
'connector.write.flush.max-rows' = '5000', -- default is 5000
'connector.write.flush.interval' = '2s', -- no default value
{code}

That means if flush interval is not set, the buffered output rows may not be flushed to database
for a long time. That is a surprising behavior because no results are outputed by default.


So we propose to have a default flush '1s' interval and '100' rows for JDBC sink flush. This
only applies to new JDBC sink options:

{code}
'sink.buffer-flush.max-rows' = '100'
'sink.buffer-flush.interval' = '1s'
{code}





  was:
Currently, JDBC sink provides 2 flush options:

{code}
'connector.write.flush.max-rows' = '5000', -- default is 5000
'connector.write.flush.interval' = '2s', -- no default value
{code}

That means if flush interval is not set, the buffered output rows may not be flushed to database
for a long time. That is a surprising behavior because no results are outputed by default.


So I propose to have a default flush '1s' interval for JDBC sink or default 1 row for flush
size. 






> Improve default flush strategy for JDBC sink to make it work out-of-box
> -----------------------------------------------------------------------
>
>                 Key: FLINK-16497
>                 URL: https://issues.apache.org/jira/browse/FLINK-16497
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / JDBC, Table SQL / Ecosystem
>            Reporter: Jark Wu
>            Assignee: Jark Wu
>            Priority: Critical
>             Fix For: 1.11.0
>
>
> Currently, old JDBC sink provides 2 flush options:
> {code}
> 'connector.write.flush.max-rows' = '5000', -- default is 5000
> 'connector.write.flush.interval' = '2s', -- no default value
> {code}
> That means if flush interval is not set, the buffered output rows may not be flushed
to database for a long time. That is a surprising behavior because no results are outputed
by default. 
> So we propose to have a default flush '1s' interval and '100' rows for JDBC sink flush.
This only applies to new JDBC sink options:
> {code}
> 'sink.buffer-flush.max-rows' = '100'
> 'sink.buffer-flush.interval' = '1s'
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message