hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
Date Fri, 02 Feb 2018 06:10:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349835#comment-16349835
] 

Steve Loughran commented on HADOOP-15204:
-----------------------------------------

Ah I see your reasoning.

Yes it would be good, but I would like it to work outside of Configuration itself; Look at
{{S3ATestUtils.getTestPropertyBytes}} as an example of a place which could adopt this.

{{Configuration.getLongBytes()}} must support the exact same set of units, returning it explicitly
as bytes. Otherwise you'd have some config options which take the new language, some old ones
which don't, and no valid explanation of why you can't say "32MB" of the "fs.s3a.blocksize"
option other than "they wrote it before the new API was added." . This particularly matters
when you start sharing properties via ${property} refs. It would not make sense for a storage
capacity to be valid in some fields, but not others


Quick code review

* IDE shuffled imports; please revert
* parseFromString() can just use Precondition.checkArgument for validation
* validation/parse errors to include value at error, and, ideally, config option too. Compare
a stack trace saying "Value not in expected format", with one saying "value of option 'buffer.size'
not in expected format "54exa"
* {{sanitizedValue.toLowerCase()}} should specify local for case conversion, same everywhere
else used.
* What if a caller doesn't want to provide a string default value of the new getters, but
just a number? That would let me return something like -1 to mean "no value set", which I
can't do with the current API.

> Add Configuration API for parsing storage sizes
> -----------------------------------------------
>
>                 Key: HADOOP-15204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15204
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 3.1.0
>            Reporter: Anu Engineer
>            Assignee: Anu Engineer
>            Priority: Minor
>             Fix For: 3.1.0
>
>         Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This JIRA proposes
to add an API like {{Configuration.getStorageSize}} which will allow users
>  to specify units like KB, MB, GB etc. This is JIRA is inspired by HADOOP-8608 and Ozone.
Adding {{getTimeDuration}} support was a great improvement for ozone code base, this JIRA
hopes to do the same thing for configs that deal with disk and memory usage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message