beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2534) KafkaIO should allow gaps in message offsets
Date Wed, 28 Jun 2017 19:33:00 GMT
Raghu Angadi created BEAM-2534:
----------------------------------

             Summary: KafkaIO should allow gaps in message offsets
                 Key: BEAM-2534
                 URL: https://issues.apache.org/jira/browse/BEAM-2534
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: Raghu Angadi
            Assignee: Raghu Angadi
            Priority: Minor
             Fix For: 2.1.0


KafkaIO reader logs a warning when it notices gaps in offsets for messages. While such gaps
are not expected for normal Kafka topics, there could be gaps when log compaction is enabled
(which deletes older messages for a key). 

This warning log is not very useful. Also we should take such gaps while estimating backlog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message