flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5075) Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite
Date Thu, 17 Nov 2016 07:33:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673047#comment-15673047

ASF GitHub Bot commented on FLINK-5075:

GitHub user tzulitai opened a pull request:


    [FLINK-5075] [kinesis] Make Kinesis consumer fail-proof to incorrect Kinesalite API behaviour

    A user reported that when tested against Kinesalite (a widely used mock Kinesis implementation),
the connector was incorrectly determining already seen shards as newly discovered.
    The problem was that the connector was using the API `describeStream(streamName, exclusiveStartIShardId)`
to fetch shards of a stream after the provided `exclusiveStartShardId` (given as the last
id of the latest shard we've already discovered), and Kinesalite behaves differently for this
from the official Kinesis API.
    For example, if the current complete shard list is [shard-0, shard-1, shard-2, shard-3]
for "test-stream",
    then `describeStream("test-stream", "shard-1")` should return: [shard-2, shard-3].
    Kinesalite, however, incorrectly returns the whole list.
    I've manually tested this change against Kinesalite, and shard discovery is working normally

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink FLINK-5075

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2822
commit cbef7eabeae645a1cd3533d3274cdb7491b1a779
Author: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
Date:   2016-11-17T06:24:24Z

    [FLINK-5075] [kinesis] Make connector fail-proof to incorrect Kinesalite API behaviour


> Kinesis consumer incorrectly determines shards as newly discovered when tested against
> -------------------------------------------------------------------------------------------------
>                 Key: FLINK-5075
>                 URL: https://issues.apache.org/jira/browse/FLINK-5075
>             Project: Flink
>          Issue Type: Bug
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
> A user reported that when our Kinesis connector is used against Kinesalite (https://github.com/mhart/kinesalite),
we're incorrectly determining already found shards as newly discovered:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Subtask-keeps-on-discovering-new-Kinesis-shard-when-using-Kinesalite-td10133.html
> I suspect the problem to be the mock Kinesis API implementations of Kinesalite doesn't
completely match with the official AWS Kinesis behaviour.

This message was sent by Atlassian JIRA

View raw message