flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5075) Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite
Date Thu, 17 Nov 2016 07:37:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673053#comment-15673053
] 

ASF GitHub Bot commented on FLINK-5075:
---------------------------------------

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2822
  
    I'm not entirely sure whether the fix is a good workaround, or whether we should really
bother with Kinesalite's incorrect behaviour at all in our Kinesis connector.
    
    I've considered the alternative is to just use the `describeStream(streamName)` variant,
always fetching the whole shard list and explicitly ruling out shards we've already seen.
That'll make the code clean of such "workarounds", but will be problematic for Kinesis users
whose shard count exceeds the largest number of shards the API can return.


> Kinesis consumer incorrectly determines shards as newly discovered when tested against
Kinesalite
> -------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-5075
>                 URL: https://issues.apache.org/jira/browse/FLINK-5075
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>
> A user reported that when our Kinesis connector is used against Kinesalite (https://github.com/mhart/kinesalite),
we're incorrectly determining already found shards as newly discovered:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Subtask-keeps-on-discovering-new-Kinesis-shard-when-using-Kinesalite-td10133.html
> I suspect the problem to be the mock Kinesis API implementations of Kinesalite doesn't
completely match with the official AWS Kinesis behaviour.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message