flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (FLINK-1935) Reimplement PersistentKafkaSource using high level Kafka API
Date Tue, 12 May 2015 21:09:01 GMT

     [ https://issues.apache.org/jira/browse/FLINK-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stephan Ewen closed FLINK-1935.
-------------------------------

> Reimplement PersistentKafkaSource using high level Kafka API
> ------------------------------------------------------------
>
>                 Key: FLINK-1935
>                 URL: https://issues.apache.org/jira/browse/FLINK-1935
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector, Streaming
>    Affects Versions: 0.9
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>             Fix For: 0.9
>
>
> The current PersistentKafkaSource in Flink has some limitations that I seek to overcome
by reimplementing it using Kafka's high level API (and manually committing the offsets to
ZK).
> This approach only works when the offsets are committed to ZK directly.
> The current PersistentKafkaSource does not integrate with existing Kafka tools (for example
for monitoring the "lag"). All the communication with Zookeeper is implemented manually in
our current code. This is prone to errors and inefficiencies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message