hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
Date Thu, 24 May 2018 10:26:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488762#comment-16488762
] 

Steve Loughran edited comment on HADOOP-15480 at 5/24/18 10:25 AM:
-------------------------------------------------------------------

some of the tests just use FileSystem.newInstance, e.g.{{ITestS3AInputStreamPerformance}}.
We don't do it everywhere as it makes for slower tests.

Looking at {{AbstractITCommitProtocol}}, it is patching in the created FS into the cache to
make sure the instance with config we want gets in...though some of the tests have asserts
to verify the FS returned is what they want

the code would be changed to something like (with the relevant config, uri)..
{code}
     S3AFileSystem fs = (S3AFileSystem) FileSystem.newInstance(uri, conf);
     S3ATestUtils.setMetadataStore(fs, new NullMetadataStore());
{code}
of course, if we are creating a new FS instance, why not just change the config to say "null
MD store", rather than dynamically patch it?


was (Author: stevel@apache.org):
some of the tests just use FileSystem.newInstance, e.g.{{ITestS3AInputStreamPerformance}}.
We don't do it everywhere as it makes for slower tests.

Looking at {{AbstractITCommitProtocol}}, it is patching in the created FS into the cache to
make sure the instance with config we want gets in...though some of the tests have asserts
to verify the FS returned is what they want

> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> -------------------------------------------------------------------
>
>                 Key: HADOOP-15480
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15480
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Gabor Bota
>            Assignee: Gabor Bota
>            Priority: Major
>         Attachments: HADOOP-15480.001.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the testDiffCommand
test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)  Time elapsed:
8.059 s  <<< FAILURE!
> java.lang.AssertionError: 
> Mismatched metadata store outputs: MS	D	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
>  expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]> 
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotEquals(Assert.java:743)
> 	at org.junit.Assert.assertEquals(Assert.java:118)
> 	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message