hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] steveloughran edited a comment on issue #666: HADOOP-16221 add option to fail operation on metadata write failure
Date Fri, 26 Apr 2019 22:35:49 GMT
steveloughran edited a comment on issue #666: HADOOP-16221 add option to fail operation on
metadata write failure
URL: https://github.com/apache/hadoop/pull/666#issuecomment-487220946
   I've just been retesting this...happy with the changes in the operation, just two things
I want to make sure we are all good with
   1. Do we need to wrap any existing IOExceptions raised in the `finishedWrite()` calls with
their own exception. I'd going to say "yes" primarily because thats how we can guarantee that
the failure won't trigger any of the retry logic used in existing operations which assume
that an IOE only ever gets raised during the main operation against S3, rather than the subsequent
metastore calls.
   2. Do we make this a new switch or bond it to auth mode? 
   (1) There's no way you'd ever want this to be disabled when in auth mode.
   (2) When not in auth mode, we are meant to be more tolerant of OOB changes in the store,
and you could consider files which have a file changed but not metastore update as "just"
an OOB update
   But in condition #2, even if we recover, there will be a period of inconsistency. Should
we silently swallow this? Or raise an exception? 
   I'm coming round to the "this will always be on unless you somehow want to disable it"
viewpoint too. Because if you aren't updating the store for some reason (example: you don't
have write perms to the table), well, that merits a failure -doesn't it?
   Accordingly, I'm going to propose
   * we do have the new config option
   * its _true_ by default. That is, unless you say otherwise, if you can't update the metastore,
it's an error.
   saying "swallow metastore updates" is a special case people should be explicitly asking
   Returning to this patch then, I'm happy with it with some small changes:
   1.  we switch the default value of `FAIL_ON_METADATA_WRITE_ERROR_DEFAULT` to true
   1. change the docs to match.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message