hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] steveloughran commented on issue #666: HADOOP-16221 add option to fail operation on metadata write failure
Date Fri, 26 Apr 2019 22:31:42 GMT
steveloughran commented on issue #666: HADOOP-16221 add option to fail operation on metadata
write failure
URL: https://github.com/apache/hadoop/pull/666#issuecomment-487220946
 
 
   I've just been retesting this...happy with the changes in the operation, just two things
I want to make sure we are all good with
   
   1. Do we need to wrap any existing IOExceptions raised in the `finishedWrite()` calls with
their own exception. I'd going to say "yes" primarily because thats how we can guarantee that
the failure won't trigger any of the retry logic used in existing operations which assume
that an IOE only ever gets raised during the main operation against S3, rather than the subsequent
metastore calls.
   
   2. Do we make this a new switch or bond it to auth mode? 
   
   (1) There's no way you'd ever want this to be disabled when in auth mode.
   (2) When not in auth mode, we are meant to be more tolerant of OOB changes in the store,
and you could consider files which have a file changed but not metastore update as "just"
an OOB update
   
   But in condition #2, even if we recover, there will be a period of inconsistency. Should
we silently swallow this? Or raise an exception? 
   
   I'm coming round to the "this will always be on unless you somehow want to disable it"
viewpoint too. Because if you aren't updating the store for some reason (example: you don't
have write perms to the table), well, that merits a failure -doesn't it?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message