hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16622) intermittent failure of ITestCommitOperations: too many s3guard writes
Date Tue, 01 Oct 2019 13:15:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941817#comment-16941817
] 

Steve Loughran commented on HADOOP-16622:
-----------------------------------------

{code}
[ERROR] testBulkCommitFiles(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)  Time elapsed:
6.264 s  <<< FAILURE!
java.lang.AssertionError: Number of records written after commit #2; first commit had 2; first
commit ancestors CommitContext{operationState=AncestorState{operation=Commitid=52; dest=s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out;
size=6; paths={s3a://hwdev-steve-ireland-new/fork-0004 s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles
s3a://hwdev-steve-ireland-new/fork-0004/test s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out
s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out/file1
s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME}}}; second commit ancestors:
CommitContext{operationState=AncestorState{operation=Commitid=52; dest=s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out;
size=8; paths={s3a://hwdev-steve-ireland-new/fork-0004 s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles
s3a://hwdev-steve-ireland-new/fork-0004/test s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir
s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out/file1
s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir/file2
s3a://hwdev-steve-ireland-new/fork-0004/test/DELAY_LISTING_ME}}}: s3guard_metadatastore_record_writes
expected:<2> but was:<7>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:932)
	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testBulkCommitFiles(ITestCommitOperations.java:636)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

{code}

sometimes this mismatch can occur when you do a getFilestatus check on a newly PUT file and
the clock is off...I thought I'd fixed that, but maybe it can still surface in overloaded
test runs

> intermittent failure of ITestCommitOperations: too many s3guard writes
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-16622
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16622
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> intermittent failure of ITestCommitOperations; expected 2 s3guard writes, saw 7
> the logged commit state shows that only two entries were added, so I'm not sure what
is up. Proposed: in HADOOP-16207 I will add s3guard.operations log to debug so we get a trace
of all DDB put/delete calls; this will let us debug it when it surfaces again



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message