hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
Date Wed, 23 May 2018 05:45:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486747#comment-16486747
] 

Aaron Fabbri commented on HADOOP-15480:
---------------------------------------

Looks good [~gabor.bota], glad the hint was helpful.

One question on this..
{noformat}
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
@@ -324,6 +324,7 @@ protected void exec(S3GuardTool cmd, ByteArrayOutputStream buf, String...args)
   @Test
   public void testDiffCommand() throws Exception {
     S3AFileSystem fs = getFileSystem();
+    S3ATestUtils.setMetadataStore(fs, new NullMetadataStore());
{noformat}
When other tests are running, and they get the same fs instance via {{FileSystem#get()}},
could this cause problems with other tests? If they happen to be doing stuff when we temporarily
null out the MetadataStore it could cause flaky tests?

I thought maybe we would need to either (1) disable cached FileSystem instances via {{S3ATestUtils.disableFilesystemCaching(conf)}},
or (2) make the S3GuardToolTestBase subclasses run in non-parallel mode by changing include/excludes
in hadoop-aws/pom.xml.  #1 would usually go in a override of getConfiguration() or getContract()
in AbstractS3GuardToolTestBase, I think.

[~stevel@apache.org] any reason we don't need to isolate this from other tests via #1 or
#2?

> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> -------------------------------------------------------------------
>
>                 Key: HADOOP-15480
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15480
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Gabor Bota
>            Assignee: Gabor Bota
>            Priority: Major
>         Attachments: HADOOP-15480.001.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the testDiffCommand
test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)  Time elapsed:
8.059 s  <<< FAILURE!
> java.lang.AssertionError: 
> Mismatched metadata store outputs: MS	D	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MS	F	100	s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MS	F	0	s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
>  expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]> 
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4, 
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotEquals(Assert.java:743)
> 	at org.junit.Assert.assertEquals(Assert.java:118)
> 	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message