hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option
Date Thu, 24 May 2018 10:38:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488778#comment-16488778
] 

Steve Loughran commented on HADOOP-14918:
-----------------------------------------

I'm not quite sure how we are meant to use this, but it broke all my other tests by deleting
the ddb table in a parallel run
{code}
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.016 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[ERROR] Tests run: 38, Failures: 0, Errors: 36, Skipped: 0, Time elapsed: 179.425 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[ERROR] testBatchWrite(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)  Time
elapsed: 100.009 s  <<< ERROR!
java.lang.Exception: test timed out after 100000 milliseconds
	at java.lang.Thread.sleep(Native Method)
	at com.amazonaws.waiters.FixedDelayStrategy.delayBeforeNextRetry(FixedDelayStrategy.java:45)
	at com.amazonaws.waiters.WaiterExecution.safeCustomDelay(WaiterExecution.java:118)
	at com.amazonaws.waiters.WaiterExecution.pollResource(WaiterExecution.java:82)
	at com.amazonaws.waiters.WaiterImpl.run(WaiterImpl.java:88)
	at com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:502)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.destroy(DynamoDBMetadataStore.java:798)
	at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.tearDown(MetadataStoreTestBase.java:129)
	at org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.tearDown(ITestDynamoDBMetadataStore.java:117)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

[ERROR] testProvisionTable(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)  Time
elapsed: 0.366 s  <<< ERROR!
java.io.FileNotFoundException: DynamoDB table 'hwdev-steve-ireland-new' is being deleted in
region eu-west-1
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:908)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:292)
	at org.apache.hadoop.fs.s3a.s3guard.MetadataStoreTestBase.setUp(MetadataStoreTestBase.java:121)
	at org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.setUp(ITestDynamoDBMetadataStore.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

> remove the Local Dynamo DB test option
> --------------------------------------
>
>                 Key: HADOOP-14918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14918
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0, 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Gabor Bota
>            Priority: Major
>         Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, HADOOP-14918-003.patch,
HADOOP-14918-004.patch
>
>
> I'm going to propose cutting out the localdynamo test option for s3guard
> * the local DDB JAR is unmaintained/lags the SDK We work with...eventually there'll be
differences in API.
> * as the local dynamo DB is unshaded. it complicates classpath setup for the build. Remove
it and there's no need to worry about versions of anything other than the shaded AWS
> * it complicates test runs. Now we need to test for both localdynamo *and* real dynamo
> * but we can't ignore real dynamo, because that's the one which matters
> While the local option promises to reduce test costs, really, it's just adding complexity.
If you are testing with s3guard, you need to have a real table to test against., And with
the exception of those people testing s3a against non-AWS, consistent endpoints, everyone
should be testing with S3Guard.
> -Straightforward to remove.-



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message