flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LiuBiao (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (FLINK-4164) Use only one rocksdb column family to store all states in an operator
Date Wed, 16 Nov 2016 03:10:58 GMT

     [ https://issues.apache.org/jira/browse/FLINK-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

LiuBiao closed FLINK-4164.
    Resolution: Won't Fix

As the state backend has been reconstructed, this issue should be closed. 

> Use only one rocksdb column family to store all states in an operator
> ---------------------------------------------------------------------
>                 Key: FLINK-4164
>                 URL: https://issues.apache.org/jira/browse/FLINK-4164
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: LiuBiao
>            Priority: Minor
> I see now in master branch, there is only one rocksdb instance in an operator, each state
will be assigned into separated column families. It's a good improvement, but I think there
are still some problems. Column families in rocksdb do not share memtable and sst files(correct
me if I am wrong, there is not much docs about column families in rocksdb). If user have thousands
of  states, the overhead will be costly. Memory may will be exhausted, also there will be
too much small sst files.
> If we use prefix in key instead of column family, the memory that rocksdb costed will
be under control. The shortcoming is that we can not optimize states separately, and performance
may be a little worse than before. But I think it is worth to do, it is meaning for some cases.

This message was sent by Atlassian JIRA

View raw message