hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] snvijaya commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
Date Mon, 30 Mar 2020 11:16:03 GMT
snvijaya commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error
URL: https://github.com/apache/hadoop/pull/1898#discussion_r400112122

 File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
 @@ -299,11 +327,32 @@ private void clearFromReadAheadQueue(final AbfsInputStream stream,
final long re
   private int getBlockFromCompletedQueue(final AbfsInputStream stream, final long position,
final int length,
-                                         final byte[] buffer) {
-    ReadBuffer buf = getFromList(completedReadList, stream, position);
-    if (buf == null || position >= buf.getOffset() + buf.getLength()) {
+                                         final byte[] buffer) throws IOException {
+    ReadBuffer buf = getBufferFromCompletedQueue(stream, position);
+    if (buf == null) {
       return 0;
+    if (buf.getStatus() == ReadBufferStatus.READ_FAILED) {
+      // Eviction of a read buffer is triggered only when a queue request comes in
+      // and each eviction attempt tries to find one eligible buffer.
+      // Hence there are chances that an old read-ahead buffer with exception is still
+      // available. To prevent new read requests to fail due to such old buffers,
+      // return exception only from buffers that failed within last THRESHOLD_AGE_MILLISECONDS
+      if ((currentTimeMillis() - (buf.getTimeStamp()) < THRESHOLD_AGE_MILLISECONDS)) {
 Review comment:
   Aim here is to enforce the read-ahead failure for the threshold time duration (which currently
is 30 sec), i.e. any read request for that offset that can be served by the ReadBuffer needs
to fail.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message