hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option
Date Fri, 25 May 2018 06:17:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16490288#comment-16490288

Aaron Fabbri commented on HADOOP-14918:

Thanks for making progress on this, [~gabor.bota], it will be great to have this fixed.

I think we should have the caller nominate a specific table to create and delete, skip the
test if unset. 
Was thinking something similar to [~stevel@apache.org] here.  I frequently run into problems
with both {{AbstractS3GuardToolTestBase#testDestroyNoBucket}} and {{ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle}}.
Not only can they nuke the table out from under other tests, but AWS's handling of table names
is very *eventually* consistent and flaky, meaning a recently re-created table can occasionally
show up as still being deleted.

Should the "destructive s3guard integration test dynamodb table" name go in the contract test

 If set, run in serial test phase (see hadoop-aws/pom.xml).  If unset, skip those tests.

> remove the Local Dynamo DB test option
> --------------------------------------
>                 Key: HADOOP-14918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14918
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0, 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Gabor Bota
>            Priority: Major
>         Attachments: HADOOP-14918-001.patch, HADOOP-14918-002.patch, HADOOP-14918-003.patch,
> I'm going to propose cutting out the localdynamo test option for s3guard
> * the local DDB JAR is unmaintained/lags the SDK We work with...eventually there'll be
differences in API.
> * as the local dynamo DB is unshaded. it complicates classpath setup for the build. Remove
it and there's no need to worry about versions of anything other than the shaded AWS
> * it complicates test runs. Now we need to test for both localdynamo *and* real dynamo
> * but we can't ignore real dynamo, because that's the one which matters
> While the local option promises to reduce test costs, really, it's just adding complexity.
If you are testing with s3guard, you need to have a real table to test against., And with
the exception of those people testing s3a against non-AWS, consistent endpoints, everyone
should be testing with S3Guard.
> -Straightforward to remove.-

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message