hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Steelman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11044) FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
Date Tue, 02 Sep 2014 18:34:23 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118486#comment-14118486
] 

Gary Steelman commented on HADOOP-11044:
----------------------------------------

[~sandyr] that's unfortunate. There's a couple ways I see to handle this then:
1) Add new methods to return long types and also store longs internally, increment them when
the ints are incremented. This requires more memory per Statistic object.
2) Add new methods to return long types and switch the internal types to long and simply cast
the internal longs to ints when returned via the int methods. Mark the int methods deprecated,
then remove them after next release. This puts overflow problem onto the int type methods.


If we go option 2, I think the API ends up cleaner in the end, but we do introduce a non-backwards
compatible change. It might be that option 1 is the best in this case.

> FileSystem counters can overflow for large number of readOps, largeReadOps, writeOps
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11044
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11044
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 2.5.0, 2.4.1
>            Reporter: Swapnil Daingade
>            Priority: Minor
>
> The org.apache.hadoop.fs.FileSystem.Statistics.StatisticsData class defines readOps,
largeReadOps, writeOps as int. Also the The org.apache.hadoop.fs.FileSystem.Statistics class
has methods like getReadOps(), getLargeReadOps() and getWriteOps() that return int. These
int values can overflow if the exceed 2^31-1 showing negative values. It would be nice if
these can be changed to long.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message