flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Richter (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-10043) Refactor object construction/inititlization/restore code
Date Fri, 03 Aug 2018 08:25:00 GMT
Stefan Richter created FLINK-10043:

             Summary: Refactor object construction/inititlization/restore code
                 Key: FLINK-10043
                 URL: https://issues.apache.org/jira/browse/FLINK-10043
             Project: Flink
          Issue Type: Sub-task
            Reporter: Stefan Richter
            Assignee: Stefan Richter

Currently, the constructor of {{RocksDBKeyedStateBackend}} has the following shortcomings:
- It does initialization and cleanup of some directories and files. this makes it harder to
unit-test because dependencies are created in the constructor and not passed in from outside.
- It leaves many important fields uninitialized and more methods e.g. {{restore}} _have_ to
be called before the object is fully constructed. This is error-prone in many ways and hard
to unit-test. I think the origin of this problem was introducing incremental snapshots, because
in this case, we can only open a RocksDB instance AFTER the restore code was executed and
restored the working directory.

As a solution, I would suggest to have a dedicated builder class that takes the current constructor
parameters and (optional) the state handles to restore. Then, this class constructs and intializes
all dependencies, and dependencies are only passed to the new {{RocksDBKeyedStateBackend}}
constructor that does no other work besided assigning dependencies to fields.

With this change, I would also extract the different restore strategies for incremental and
full snapshots out of the main class, into their own classes. They will then be used in the
newly introduced builder from the previous step.

This message was sent by Atlassian JIRA

View raw message