hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] steveloughran commented on a change in pull request #768: HADOOP-16269. ABFS: add listFileStatus with StartFrom.
Date Fri, 26 Apr 2019 15:31:22 GMT
steveloughran commented on a change in pull request #768: HADOOP-16269. ABFS: add listFileStatus
with StartFrom.
URL: https://github.com/apache/hadoop/pull/768#discussion_r278999013
 
 

 ##########
 File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
 ##########
 @@ -583,6 +620,69 @@ public FileStatus getFileStatus(final Path path) throws IOException
{
     return fileStatuses.toArray(new FileStatus[fileStatuses.size()]);
   }
 
+  // generate continuation token for xns account
+  private String generateContinuationTokenForXns(final String firstEntryName) {
+    if (firstEntryName == null || firstEntryName.isEmpty() || firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH))
{
+      throw new IllegalArgumentException("startFrom must be a dir/file name and it can not
be a full path");
+    }
+
+    StringBuilder sb = new StringBuilder();
+    sb.append(firstEntryName).append("#$").append("0");
+
+    CRC64 crc64 = new CRC64();
+    StringBuilder token = new StringBuilder();
+    token.append(crc64.compute(sb.toString().getBytes(StandardCharsets.UTF_8)))
+            .append(SINGLE_WHITE_SPACE)
+            .append("0")
+            .append(SINGLE_WHITE_SPACE)
+            .append(firstEntryName);
+
+    return Base64.encode(token.toString().getBytes(StandardCharsets.UTF_8));
+  }
+
+  // generate continuation token for non-xns account
+  private String generateContinuationTokenForNonXns(final String path, final String firstEntryName)
{
+    if (firstEntryName == null || firstEntryName.isEmpty() || firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH))
{
+      throw new IllegalArgumentException("startFrom must be a dir/file name and it can not
be a full path");
+    }
+
+    // Notice: non-xns continuation token requires full path (first "/" is not included)
for startFrom
+    final String startFrom = (path.isEmpty() || path.equals(ROOT_PATH))
+            ? firstEntryName
+            : path + ROOT_PATH + firstEntryName;
+
+    SimpleDateFormat simpleDateFormat = new SimpleDateFormat(TOKEN_DATE_PATTERN, Locale.US);
+    String date = simpleDateFormat.format(new Date());
+
+    StringBuilder sb = new StringBuilder(path.length() + startFrom.length() + date.length()
+ 24);
+    sb.append(String.format("%06d", path.length())).append(CHAR_EXCLAMATION_POINT)
 
 Review comment:
   any reason not to use a single large String.format? I'm just wondering which version is
the most clear as to what is being constructed. If the CHAR_EXCLAMATION_POINT was just "!"
inlined the case for a single String.format would be clear. Here: not so sure

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message