tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3877) Delete spill files once merge is done
Date Fri, 15 Dec 2017 20:26:00 GMT
Rohini Palaniswamy created TEZ-3877:

             Summary: Delete spill files once merge is done
                 Key: TEZ-3877
                 URL: https://issues.apache.org/jira/browse/TEZ-3877
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Rohini Palaniswamy

  I see that spill files are not deleted right after merge completes. We should do that as
it takes up a lot of space and we can't afford that wastage when Tez takes up a lot of shuffle
space with complex DAGs. [~jlowe] told me they are only cleaned up after application completes
as they are written in app directory and not container directory. That also has to be done
so that they are cleaned up by node manager during task failures or container crashes.

This message was sent by Atlassian JIRA

View raw message