hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] steveloughran commented on a change in pull request #802: HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries …
Date Fri, 31 May 2019 19:31:47 GMT
steveloughran commented on a change in pull request #802: HADOOP-16279. S3Guard: Implement
time-based (TTL) expiry for entries …
URL: https://github.com/apache/hadoop/pull/802#discussion_r289524041
 
 

 ##########
 File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
 ##########
 @@ -271,6 +274,92 @@ public void testListingDelete() throws Exception {
     deleteFileInListing();
   }
 
+  /**
+   * Tests that tombstone expiry is implemented, so if a file is created raw
+   * while the tombstone exist in ms for with the same name then S3Guard will
+   * check S3 for the file.
+   *
+   * Seq: create guarded; delete guarded; create raw (same path); read guarded;
+   * This will fail if no tombstone expiry is set
+   *
+   * @throws Exception
+   */
+  @Test
+  public void testTombstoneExpiryGuardedDeleteRawCreate() throws Exception {
+    boolean allowAuthoritative = authoritative;
+    Path testFilePath = path("TEGDRC-" + UUID.randomUUID());
+    LOG.info("Allow authoritative param: {}",  allowAuthoritative);
+    String originalText = "some test";
+    String newText = "the new originalText for test";
+
+    final ITtlTimeProvider originalTimeProvider =
+        guardedFs.getTtlTimeProvider();
+    try {
+      final AtomicLong now = new AtomicLong(1);
+      final AtomicLong metadataTtl = new AtomicLong(1);
+      // todo add ttl for entries
+      // final AtomicLong entriesTtl = new AtomicLong(1);
+
+      // SET TTL TIME PROVIDER FOR TESTING
+      ITtlTimeProvider testTimeProvider =
+          new ITtlTimeProvider() {
+            @Override public long getNow() {
+              return now.get();
+            }
+
+            @Override public long getMetadataTtl() {
+              return metadataTtl.get();
+            }
+          };
+      guardedFs.setTtlTimeProvider(testTimeProvider);
+
+      // CREATE GUARDED
+      writeTextFile(guardedFs, testFilePath, originalText, true);
+      // wait until the file is created
+      final FileStatus origStatus = awaitFileStatus(rawFS, testFilePath);
+      assertNotNull("Create file status should not be null", origStatus);
+
+      // DELETE GUARDED
+      guardedFs.delete(testFilePath, true);
+
+
+      final PathMetadata metadata =
+          guardedFs.getMetadataStore().get(testFilePath);
+      assertNotNull("Created file metadata should not be null in ms",
+          metadata);
+      assertEquals("Created file metadata should equal with mocked now",
+          now.get(), metadata.getLastUpdated());
+
+      // CREATE RAW
+      writeTextFile(rawFS, testFilePath, newText, true);
+      final FileStatus newStatus = awaitFileStatus(rawFS, testFilePath);
+      assertNotNull("Newly created file status should not be null.",
+          newStatus);
+
+      // CHANGE TTL SO ENTRY (& TOMBSTONE METADATA) WILL EXPIRE
+      long willExpire = now.get() + metadataTtl.get() + 1l;
 
 Review comment:
   nit: use 1L

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message