beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu Mingmin (JIRA)" <>
Subject [jira] [Commented] (BEAM-1775) fix issue of start_from_previous_offset in KafkaIO
Date Wed, 22 Mar 2017 05:33:41 GMT


Xu Mingmin commented on BEAM-1775:

Hi [], 
1). for KafkaCheckpointMark and Beam state, I totally agree that's the right way to go;
2). my question mostly goes when user don't enable checkpoint for any reason with KafkaIO.

In Kafka, 'earliest' means the beginning of a topic, 'latest' means the end of a topic. I
think what users want is to 'consume from the offset of last run'. As Kafka 0.9+ can manage
consumer offset, it's possible to support this feature in KafkaIO. 

> fix issue of start_from_previous_offset in KafkaIO
> --------------------------------------------------
>                 Key: BEAM-1775
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>            Reporter: Xu Mingmin
>            Assignee: Xu Mingmin
> Jins George via 
> 5:50 PM (15 hours ago)
> to user
> Hello,
> I am writing a Beam pipeline(streaming) with Flink runner to consume data from Kafka
and apply some transformations and persist to Hbase.
> If I restart the application ( due to failure/manual restart), consumer does not resume
from the offset where it was prior to restart. It always resume from the latest offset.
> If I enable Flink checkpionting with hdfs state back-end, system appears to be resuming
from the earliest offset
> Is there a recommended way to resume from the offset where it was stopped ?
> Thanks,
> Jins George

This message was sent by Atlassian JIRA

View raw message