hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Swapnil Daingade (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11044) FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
Date Thu, 04 Sep 2014 18:40:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121739#comment-14121739
] 

Swapnil Daingade commented on HADOOP-11044:
-------------------------------------------

Looked at the test failures. I am not sure if these are directly related to the fix. Will
investigate more. Is it possible that these were due to some intermittent issues?
Should I submit the same patch again? Wanted to check before I did as I don't want to consume
resources

* org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode
java.io.IOException: All datanodes 127.0.0.1:48517 are bad. Aborting...
	at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:343)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathOutputStreamRunner$1.close(WebHdfsFileSystem.java:776)
	at org.apache.hadoop.hdfs.AppendTestUtil.testAppend(AppendTestUtil.java:198)
	at org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract.testResponseCode(TestWebHdfsFileSystemContract.java:461)

* org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testPipelineRecoveryStress
java.lang.RuntimeException: Deferred
	at org.apache.hadoop.test.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:130)
	at org.apache.hadoop.test.MultithreadedTestUtil$TestContext.waitFor(MultithreadedTestUtil.java:121)
	at org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testPipelineRecoveryStress(TestPipelinesFailover.java:485)
Caused by: org.apache.hadoop.ipc.RemoteException: File /test-21 could only be replicated to
0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are
excluded in this operation.


> FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11044
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11044
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 2.5.0, 2.4.1
>            Reporter: Swapnil Daingade
>            Priority: Minor
>         Attachments: 11044.patch3
>
>
> The org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData class defines readOps,
largeReadOps, writeOps as int. Also the The org.apache.hadoop.fs.FileSystem.Statistics class
has methods like getReadOps(), getLargeReadOps() and getWriteOps() that return int. These
int values can overflow if the exceed 2^31-1 showing negative values. It would be nice if
these can be changed to long.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message