tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [tez] toddlipcon opened a new pull request #40: TEZ-1348. Fix local mode with non-local default FS
Date Thu, 04 Apr 2019 05:24:58 GMT
toddlipcon opened a new pull request #40: TEZ-1348. Fix local mode with non-local default FS
URL: https://github.com/apache/tez/pull/40
 
 
   This fixes local mode to be able to run when fs.defaultFS is not local.
   The main change is that, when setting up the working directory for the
   AM, we need to consider that the staging directory may not be on the
   local file system, and appropriately construct the right FileSystem
   instances to copy the data into the working directory. This also fixes
   the loading of the conf PB file to come from that same local working
   directory instead of attempting to read it off the staging directory.
   
   Along the way I did a bit of cleanup:
   - simplified TezUtilsInternal.readUserSpecifiedTezConfiguration to use a
     try-with-resources and to throw an exception instead of returning null
     if the file doesn't exist. All of the call sites of this method
     assumed a non-null return value, so any case that it would have
     returned null previously would have resulted in an NPE anyway.
   
   - removed a commented-out variant of the same method
   
   - removed LocalClient.getPluginDescriptorInfo. This had some unnecessary
     complexity and checks and the necessary code takes just one line at
     the call site.
   
   To test, I modified the existing TestLocalMode to parameterize it: it
   now runs once with fs.defaultFS as the local filesystem, and once with
   it as HDFS. The modified test fails prior to the patch and passes now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message