flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Bhushan Ratnakar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-10536) Flink Kinesis Consumer leading to job failure due to ProvisionedThroughputExceededException
Date Fri, 12 Oct 2018 07:25:00 GMT
Ravi Bhushan Ratnakar created FLINK-10536:
---------------------------------------------

             Summary: Flink Kinesis Consumer leading to job failure due to ProvisionedThroughputExceededException
                 Key: FLINK-10536
                 URL: https://issues.apache.org/jira/browse/FLINK-10536
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
    Affects Versions: 1.5.2
            Reporter: Ravi Bhushan Ratnakar


*Background:*

Flink Kinesis consumer, while consuming message from a shard, it retries for a configured
number of attempts and then after it throws "ProvisionedThroughputExceededException" and which
bubbles up to top level and eventually which leads to failure of the running streaming job.

*Improvement:*

In some scenario it may be desirable that to skip some of the records by skipping that "shardIterator"
and move to next iterator in "ShardConsumer" and log this as metrics. This can be configurable
through some property. This way those application which can tolerate to skip some messages
can continue to keep running.

 

If this idea seems useful, i would like to contribute on this. Please let me know the process

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message