tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3237) Corrupted shuffle transfers to disk are not detected during transfer
Date Fri, 29 Apr 2016 15:24:12 GMT
Jason Lowe created TEZ-3237:
-------------------------------

             Summary: Corrupted shuffle transfers to disk are not detected during transfer
                 Key: TEZ-3237
                 URL: https://issues.apache.org/jira/browse/TEZ-3237
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.7.0
            Reporter: Jason Lowe


When a shuffle transfer is larger than the single transfer limit it gets written straight
to disk during the transfer.  Unfortunately there are no checksum validations performed during
that transfer, so if the data is corrupted at the source or during transmit it goes undetected.
 Only later when the task tries to consume the transferred data is the error detected, but
at that point it's too late to blame the source task for the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message