flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [flink] haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE)
Date Wed, 01 May 2019 09:03:41 GMT
haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support
for checkpointing (ATLEAST_ONCE)
URL: https://github.com/apache/flink/pull/6594#issuecomment-488239680
   The checkpoints take e.g. `12528 ms` to write, but they are not large at all:
   <img width="395" alt="Screenshot 2019-05-01 at 11 01 51" src="https://user-images.githubusercontent.com/193115/57011163-8c3b3100-6c00-11e9-88de-5e3f59086ab5.png">
   I put the checkpoint period to:
           env.enableCheckpointing(9000); // checkpoint every N seconds
   But I've tried 1000 and 1500 respectively, as well with approximately the same results.
   The creation looks like this:
           final String subscriptionName = params.get("pubsub-subscription", Env.getOrElse("ANALYTICS_PUBSUB_SUBSCRIPTION",
           String jsonFilePath =
           if (! new File(jsonFilePath).exists()) {
               throw new Exception(String.format("Couldn't find path: '%s'", jsonFilePath));
           Credentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonFilePath));
           logger.info("Using Credential JSON: {}, subscription name: {}", jsonFilePath, subscriptionName);
           PubSubSource<AppEvent> source = PubSubSource.newBuilder(new AppEventSerializer(),
projectId, subscriptionName).withCredentials(credentials).build();
           return env.addSource(source).uid(String.format("pubsub-%s", subscriptionName));

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

View raw message