hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15837) DynamoDB table Update can fail S3A FS init
Date Wed, 10 Oct 2018 14:21:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645031#comment-16645031
] 

Steve Loughran commented on HADOOP-15837:
-----------------------------------------

Patch 002; fix checkstyle. 

Tested S3 ireland with " -Ds3guard -Ddynamodb"; failures in unrelated issues as discussed
and covered elsewhere

* this patch is ready for review
* I plan to backport to 2.10-3.9 the change to the case statement as that addresses the key
issue of capacity change while S3Guard is in use, which autoscale will implicitly do in high-load
situations (i.e. use S3Guard heavily, DDB triggers scale up, S3Guard fails)

+ [~mackrorysd] : can you look @ this. It's a serious issue which will surface in the wild

Errors
{code}
[ERROR] Errors: 
[ERROR]   ITestS3GuardConcurrentOps.testConcurrentTableCreations:166->deleteTable:77 »
 ...
[ERROR]   ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle:315 » ResourceInUse Atte...
[ERROR]   ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testSetCapacityFailFastIfNotGuarded:330->AbstractS3GuardToolTestBase.lambda$testSetCapacityFailFastIfNotGuarded$2:331->AbstractS3GuardToolTestBase.run:115
» FileNotFound
{code}

> DynamoDB table Update can fail S3A FS init
> ------------------------------------------
>
>                 Key: HADOOP-15837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15837
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.2.0
>         Environment: s3guard test with small capacity (10) but autoscale enabled &
multiple consecutive parallel test runs executed...this seems to have been enough load to
trigger the state change
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15837-001.patch, HADOOP-15837-002.patch
>
>
> When DDB autoscales a table, it goes into an UPDATING state. The waitForTableActive operation
in the AWS SDK doesn't seem to wait long enough for this to recover. We need to catch &
retry



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message