flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [flink] haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE)
Date Wed, 01 May 2019 09:03:41 GMT
haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support
for checkpointing (ATLEAST_ONCE)
URL: https://github.com/apache/flink/pull/6594#issuecomment-488239680
 
 
   The checkpoints take e.g. `12528 ms` to write, but they are not large at all:
   
   <img width="395" alt="Screenshot 2019-05-01 at 11 01 51" src="https://user-images.githubusercontent.com/193115/57011163-8c3b3100-6c00-11e9-88de-5e3f59086ab5.png">
   
   I put the checkpoint period to:
   
   ```
           env.enableCheckpointing(9000); // checkpoint every N seconds
           env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2000);
   ```
   
   But I've tried 1000 and 1500 respectively, as well with approximately the same results.
   
   The creation looks like this:
   
   ```java
           final String subscriptionName = params.get("pubsub-subscription", Env.getOrElse("ANALYTICS_PUBSUB_SUBSCRIPTION",
"logs-analytics-dev"));
   
           String jsonFilePath =
               Optional
                   .ofNullable(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))
                   .orElse("./analytics-dev.json");
   
           if (! new File(jsonFilePath).exists()) {
               throw new Exception(String.format("Couldn't find path: '%s'", jsonFilePath));
           }
           Credentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonFilePath));
           logger.info("Using Credential JSON: {}, subscription name: {}", jsonFilePath, subscriptionName);
   
           PubSubSource<AppEvent> source = PubSubSource.newBuilder(new AppEventSerializer(),
projectId, subscriptionName).withCredentials(credentials).build();
           return env.addSource(source).uid(String.format("pubsub-%s", subscriptionName));
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message