hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11601) Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files
Date Mon, 16 May 2016 17:56:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284942#comment-15284942

Daniel Templeton commented on HADOOP-11601:

Thanks, for the patch [~steve_l].

My comments:

-that is too small will result in either too many jobs being submitted
-for efficient work, or the `JobSubmissionClient` running out of memory.
+that is too small will result in badly partitioned workload.

Your change here correctly states the root effect, whereas what you're removing states the
visible side effects.  Seems to me that they're both useful.  Maybe add your statement with
the existing statement as a clarifying subordinate phrase?

        assertPathExists("expected path to be visible before anything written",

I don't understand the message here.  Didn't you just write 'a' and then flush it?

    assertTrue("fs.getDefaultBlockSize(path) size is invalid " + defaultBlockSize,
        defaultBlockSize > 0);

Would it be helpful here to replace {{path}} with the actual path?

   * @param parent Test file parent dir path
   *    thrown if an I/O error occurs while writing or reading the test file
   *    <i>other than file not found</i>
   * @return the status
   * @throws IOException

The throws message does not follow the {{@throws}} tag.

In the {{getFileStatusEventually()}} method, would it be better to include a short (~10ms)
sleep to reduce the impact during parallel testing?

> Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty
> -----------------------------------------------------------------------------------
>                 Key: HADOOP-11601
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11601
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, test
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HADOOP-11601-001.patch, HADOOP-11601-002.patch
>   Original Estimate: 1h
>  Remaining Estimate: 1h
> HADOOP-11584 has shown that the contract tests are not validating that {{FileStatus.getBlocksize()}}
must be >0 for any analytics jobs to partition workload correctly. 
> Clarify in text and add test to do this. Test MUST be designed to work against eventually
consistent filesystems where {{getFileStatus()}} may not be immediately visible, by retrying
operation if FS declares it is an object store.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message