flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5075) Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite
Date Tue, 22 Nov 2016 13:59:59 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686823#comment-15686823
] 

ASF GitHub Bot commented on FLINK-5075:
---------------------------------------

Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2822
  
    Thanks for the review @StephanEwen. I'm pretty sure this doesn't affect the normal Kinesis
shard discovery. I'll give it some final tests before merging (would like to get this in before
the next 1.1.4 RC).
    
    Yes, using `describeStream(streamName)` only will be problematic for users with large
numbers of shards, because the whole list may not be able to be returned in a single call.
So that's most likely not a solution we can consider.


> Kinesis consumer incorrectly determines shards as newly discovered when tested against
Kinesalite
> -------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-5075
>                 URL: https://issues.apache.org/jira/browse/FLINK-5075
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>
> A user reported that when our Kinesis connector is used against Kinesalite (https://github.com/mhart/kinesalite),
we're incorrectly determining already found shards as newly discovered:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Subtask-keeps-on-discovering-new-Kinesis-shard-when-using-Kinesalite-td10133.html
> I suspect the problem to be the mock Kinesis API implementations of Kinesalite doesn't
completely match with the official AWS Kinesis behaviour.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message