beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Groh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-1659) Improve UnboundedSourceShard structure
Date Wed, 08 Mar 2017 19:35:45 GMT
Thomas Groh created BEAM-1659:
---------------------------------

             Summary: Improve UnboundedSourceShard structure
                 Key: BEAM-1659
                 URL: https://issues.apache.org/jira/browse/BEAM-1659
             Project: Beam
          Issue Type: Improvement
          Components: runner-direct
            Reporter: Thomas Groh
            Priority: Minor


Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how
to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints
that can be resumed from can also be finalized, but otherwise the two are unrelated. This
also improves ease of never multi-finalizing a checkpoint. This gets more complicated with
retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory
handle retries properly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message