flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1935) Reimplement PersistentKafkaSource using high level Kafka API
Date Thu, 23 Apr 2015 12:52:50 GMT
Robert Metzger created FLINK-1935:

             Summary: Reimplement PersistentKafkaSource using high level Kafka API
                 Key: FLINK-1935
                 URL: https://issues.apache.org/jira/browse/FLINK-1935
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector, Streaming
    Affects Versions: 0.9
            Reporter: Robert Metzger
            Assignee: Robert Metzger
             Fix For: 0.9

The current PersistentKafkaSource in Flink has some limitations that I seek to overcome by
reimplementing it using Kafka's high level API (and manually committing the offsets to ZK).

This approach only works when the offsets are committed to ZK directly.

The current PersistentKafkaSource does not integrate with existing Kafka tools (for example
for monitoring the "lag"). All the communication with Zookeeper is implemented manually in
our current code. This is prone to errors and inefficiencies.

This message was sent by Atlassian JIRA

View raw message