flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-8001) Mark Kafka Consumer as idle if it doesn't have partitions
Date Mon, 06 Nov 2017 18:44:01 GMT
Aljoscha Krettek created FLINK-8001:
---------------------------------------

             Summary: Mark Kafka Consumer as idle if it doesn't have partitions
                 Key: FLINK-8001
                 URL: https://issues.apache.org/jira/browse/FLINK-8001
             Project: Flink
          Issue Type: Bug
            Reporter: Aljoscha Krettek
            Priority: Blocker
             Fix For: 1.4.0, 1.3.3


In Flink 1.3.x the Kafka Consumer will emit a {{Long.MAX_VALUE}} watermark if it has zero
partitions assigned. If this happens and other parallel instances of the Kafka Consumer are
marked as idle (which currently never happens by default but does happen in custom forks of
our Kafka code) this means that the watermark jumps to {{Long.MAX_VALUE}} downstream.

In Flink 1.4.x this happens implicitly in the {{PeriodicWatermarkEmitter}} in {{AbstractFetcher}}
where the watermark is {{Long.MAX_VALUE}} if we don't have any partitions. This should be
changed to mark the source as idle instead, if we don't have any partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message