flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhijiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-5138) Add the method in state backend to estimate memory by hint of state size in ResourceSpec
Date Tue, 07 Mar 2017 09:35:32 GMT

     [ https://issues.apache.org/jira/browse/FLINK-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zhijiang updated FLINK-5138:
----------------------------
    Description: 
Users may specify the state size in *setResources* API, then the state backend would roughly
estimate the different kinds of memory usages based on the state size.  Currently the specific
state backend ({{FsStateBackend}}, {{MemoryStateBackend}}, etc) can implement the default
unknown resource for it to work around. 

There are two advantages to do this:

  - This estimate memory will be considered together when requesting resource for container,
so the total memory usage will not exceed the container limit.
  - Especially for *RocksDB* backend, the proper memory setting for *RocksDB* may get better
performance in  read and write operation.

  was:
Users may specify the state size in *setResources* API, then the state backend would roughly
estimate the different kinds of memory usages based on the state size. This part of estimate
memory would be aggregated with other memories in *ResourceSpec*. 

There are two advantages to do this:

  - For *RocksDB* backend, the proper memory setting for *RocksDB* may get better performance
in  read and write operation.
  - This estimate memory will be considered together when requesting resource for container,
so the total memory usage will not exceed the container limit.


> Add the method in state backend to estimate memory by hint of state size in ResourceSpec
> ----------------------------------------------------------------------------------------
>
>                 Key: FLINK-5138
>                 URL: https://issues.apache.org/jira/browse/FLINK-5138
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: zhijiang
>            Assignee: zhijiang
>
> Users may specify the state size in *setResources* API, then the state backend would
roughly estimate the different kinds of memory usages based on the state size.  Currently
the specific state backend ({{FsStateBackend}}, {{MemoryStateBackend}}, etc) can implement
the default unknown resource for it to work around. 
> There are two advantages to do this:
>   - This estimate memory will be considered together when requesting resource for container,
so the total memory usage will not exceed the container limit.
>   - Especially for *RocksDB* backend, the proper memory setting for *RocksDB* may get
better performance in  read and write operation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message