flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jgrier <...@git.apache.org>
Subject [GitHub] flink pull request #6290: [Flink-9691] [Kinesis Connector] Attempt to call g...
Date Tue, 10 Jul 2018 00:17:57 GMT
GitHub user jgrier opened a pull request:


    [Flink-9691]  [Kinesis Connector] Attempt to call getRecords() at correct frequency.

    ## What is the purpose of the change
    The purpose of this change is to make the Kinesis connector call getRecords() on the interval
specified by the SHARD_GETRECORDS_INTERVAL_MILLIS configuration parameter.  Without this change
the Kinesis Connector sleeps() for the given interval regardless of how long it takes to process
the records.  This hurts badly when trying read through a backlog of data for example.  With
this change we instead time the loop and sleep() for the specified interval MINUS any time
spent processing.  This is more in line with what user's expect and makes the connector perform
much better when there is a backlog of data to read through.
    ## Brief change log
    Time the run loop and subtract that time from the time spent sleeping.
    ## Verifying this change
    This change is a trivial rework / code cleanup without any test coverage.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): no
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: no
      - The runtime per-record code paths (performance sensitive): no
      - Anything that affects deployment or recovery: no
      - The S3 file system connector: no
    ## Documentation
      - Does this pull request introduce a new feature? no

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lyft/flink flink-9691-fix-runloop

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6290
commit 93028c19707776bc6af636e308f9b4763e2a727b
Author: Jamie Grier <jgrier@...>
Date:   2018-07-09T21:20:47Z

    Modify runloop to try to track a particular getRecords() frequency.

commit 580c4001bc60cf4d92815047697a47bc6de96cf8
Author: Jamie Grier <jgrier@...>
Date:   2018-07-09T21:41:59Z

    Fix indentation.



View raw message