flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7366) Upgrade kinesis producer library in flink-connector-kinesis
Date Tue, 29 Aug 2017 06:07:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144795#comment-16144795

ASF GitHub Bot commented on FLINK-7366:

Github user bowenli86 commented on the issue:

    can anyone from data artisan take a look at this PR please?

> Upgrade kinesis producer library in flink-connector-kinesis
> -----------------------------------------------------------
>                 Key: FLINK-7366
>                 URL: https://issues.apache.org/jira/browse/FLINK-7366
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>    Affects Versions: 1.3.2
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>             Fix For: 1.4.0, 1.3.3
> We need to upgrade KPL and KCL to pick up the enhanced performance and stability for
Flink to work better  with Kinesis. Upgrading KPL is specially necessary, because the KPL
version Flink uses is old, and doesn't have good retry and error handling logic.
> *KPL:*
> flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which is released
in Nov 2015 by AWS. It's old. It's the fourth release, and thus problematic. It doesn't even
have good retry logic, therefore Flink fails really frequently (about every 10 mins as we
observed) when Flink writes too fast to Kinesis and receives RateLimitExceededException, 
> Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56, "*With the
newer version of the KPL it uses the AWS C++ SDK which should offer additional retries.*"
on Oct 2016. 0.12.5, the version we are upgrading to, is released in May 2017 and should have
the enhanced retry logic.
> *KCL:*
> Upgrade KCL from 1.6.2 to 1.8.1
> from 1.10.71 to 1.11.171

This message was sent by Atlassian JIRA

View raw message