flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shixiaogang <...@git.apache.org>
Subject [GitHub] flink issue #3359: [FLINK-5544][streaming] Add InternalTimerService implemen...
Date Wed, 29 Nov 2017 10:05:27 GMT
Github user shixiaogang commented on the issue:

    https://github.com/apache/flink/pull/3359
  
    Very sorry for the delay. I was engaged at the work in the past months, making flink capable
of the terrible data flows in Singles Day. 
    
    RocksDBInternalTimerService is among the improvements done. 
    But we adopt a very different implementation since the initial implementation presented
here has several problems:
    * The initial implementation requires other rocksdb instances than the one used in RocksDBKeyedStateBackend,
which makes the resource configuration very difficult. 
    * The snapshotting of RocksDBInternalTimerService here is very inefficient. Though an
asynchronous and incremental implementation is available, it will duplicate much code in RocksDBKeyedStateBackend.
    
    We address these problem by introducing `SecondaryKeyedState`s which provide non-keyed
access methods to the data inside a key group. Similar to normal keyed state, secondary keyed
states are partitioned in to key groups and are also stored in the backends. Hence these secondary
states can also benefit from asynchronous and incremental snapshotting in `RocksDBKeyedStateBackend`.

    
    What do you think of the changes ? @StefanRRichter 
    
     


---

Mime
View raw message