flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StephanEwen <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-2974] Add periodic offset committer for...
Date Fri, 20 Nov 2015 11:55:36 GMT
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1341#discussion_r45461644
  
    --- Diff: flink-streaming-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
---
    @@ -374,12 +372,33 @@ public void open(Configuration parameters) throws Exception {
     			// no restore request. Let the offset handler take care of the initial offset seeking
     			offsetHandler.seekFetcherToInitialOffsets(subscribedPartitions, fetcher);
     		}
    +
    +
     	}
     
     	@Override
     	public void run(SourceContext<T> sourceContext) throws Exception {
     		if (fetcher != null) {
    +			// For non-checkpointed sources, a thread which periodically commits the current offset
into ZK.
    +			PeriodicOffsetCommitter offsetCommitter = null;
    +
    +			// check whether we need to start the periodic checkpoint committer
    +			StreamingRuntimeContext streamingRuntimeContext = (StreamingRuntimeContext) getRuntimeContext();
    +			if (!streamingRuntimeContext.isCheckpointingEnabled()) {
    +				// we use Kafka's own configuration parameter key for this.
    +				// Note that the default configuration value in Kafka is 60 * 1000, so we use the
    +				// same here.
    +				long commitInterval = Long.valueOf(props.getProperty("auto.commit.interval.ms", "60000"));
    +				offsetCommitter = new PeriodicOffsetCommitter(commitInterval, this);
    +				offsetCommitter.start();
    +				LOG.info("Starting periodic offset committer, with commit interval of {}ms", commitInterval);
    +			}
    +
     			fetcher.run(sourceContext, valueDeserializer, lastOffsets);
    +
    +			if(offsetCommitter != null) {
    --- End diff --
    
    Style space ;-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message