hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12994) Specify PositionedReadable, add contract tests, fix problems
Date Thu, 07 Apr 2016 21:14:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231078#comment-15231078

Chris Nauroth commented on HADOOP-12994:

[~stevel@apache.org], unfortunately the Azure test still failed.  Sorry, I really should have
given you the full stack trace the first time.  See below.

Your latest patch changed the single-byte read, but this test failure actually happens within
the bulk positional read.  It's the very last assertion of {{testReadSmallFile}}.

{{NativeAzureFsInputStream}} does not override the positional read method, so it inherits
the seek/bulk read/seek back implementation from {{FSInputStream}}.  {{NativeAzureFsInputStream#seek}}
is coded to raise the EOF eagerly, so this is where it fails.  When I fixed the test locally,
I did so by copy-pasting your override of the positional read method from {{S3AInputStream}}.

Your comment about covering this logic in the base class seems apt considering the copy-pasting
to different subclasses.

testReadSmallFile(org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek)  Time elapsed:
4.038 sec  <<< ERROR!
java.io.EOFException: Attempted to seek or read past the end of the file
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.seek(NativeAzureFileSystem.java:833)
	at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:70)
	at org.apache.hadoop.fs.BufferedFSInputStream.read(BufferedFSInputStream.java:108)
	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
	at org.apache.hadoop.fs.contract.AbstractContractSeekTest.testReadSmallFile(AbstractContractSeekTest.java:568)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

> Specify PositionedReadable, add contract tests, fix problems
> ------------------------------------------------------------
>                 Key: HADOOP-12994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12994
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-12994-001.patch, HADOOP-12994-002.patch, HADOOP-12994-003.patch,
> Some work on S3a has shown up that there aren't tests catching regressions in readFully,
reviewing the documentation shows that its specification could be improved.
> # review the spec
> # review the implementations
> # add tests (proposed: to the seek contract; streams which support seek should support
positioned readable)
> # fix code, where it differs significantly from HDFS or LocalFS

This message was sent by Atlassian JIRA

View raw message