hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13028) add low level counter metrics for S3A; use in read performance tests
Date Fri, 06 May 2016 18:50:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274562#comment-15274562

Chris Nauroth commented on HADOOP-13028:

Patch v010 addresses all of my prior feedback.

It's fine to keep {{S3AInstrumentation#gauge}}.

I agree that "must be private and have accessor methods" is pedantic.  I think we can control
this with the Checkstyle [HiddenField|http://checkstyle.sourceforge.net/config_coding.html#HiddenField]
rule settings.  Let's not bother for this patch though.  You've already done a lot to clean
up Checkstyle warnings.

I agree with updating the JIRA title and closing down the other ones that were folded into
this patch.

I'm now seeing a failure in {{TestS3ABlocksize#testBlockSize}}:
testBlockSize(org.apache.hadoop.fs.s3a.TestS3ABlocksize)  Time elapsed: 3.17 sec  <<<
java.lang.AssertionError: Double default block size in stat(): S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/test/testBlockSize/file;
isDirectory=false; length=1024; replication=1; blocksize=33554432; modification_time=1462559933000;
access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} expected:<67108864>
but was:<33554432>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.apache.hadoop.fs.s3a.TestS3ABlocksize.testBlockSize(TestS3ABlocksize.java:65)

It looks like the test has an expectation that you can update block size in the {{Configuration}}
of an already-initialized {{S3AFileSystem}}, and it will start using the new value.  After
this patch, the block size is read from {{Configuration}} once and cached.  I think the expectation
of the test is a little dubious, but that's the behavior that shipped in 2.7.0.  It's probably
safest to remove this part of the patch for now.

I just have one more request.  Please remove the unused import of {{Configuration}} in {{S3AInstrumentation}}.

I'll be +1 after that.  We'll also need a separate trunk patch.

> add low level counter metrics for S3A; use in read performance tests
> --------------------------------------------------------------------
>                 Key: HADOOP-13028
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13028
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, metrics
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch, HADOOP-13028-004.patch,
HADOOP-13028-005.patch, HADOOP-13028-006.patch, HADOOP-13028-007.patch, HADOOP-13028-008.patch,
HADOOP-13028-009.patch, HADOOP-13028-branch-2-008.patch, HADOOP-13028-branch-2-009.patch,
HADOOP-13028-branch-2-010.patch, org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt,
> against S3 (and other object stores), opening connections can be expensive, closing connections
may be expensive (a sign of a regression). 
> S3A FS and individual input streams should have counters of the # of open/close/failure+reconnect
operations, timers of how long things take. This can be used downstream to measure efficiency
of the code (how often connections are being made), connection reliability, etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message