beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Groh (JIRA)" <>
Subject [jira] [Created] (BEAM-1659) Improve UnboundedSourceShard structure
Date Wed, 08 Mar 2017 19:35:45 GMT
Thomas Groh created BEAM-1659:

             Summary: Improve UnboundedSourceShard structure
                 Key: BEAM-1659
             Project: Beam
          Issue Type: Improvement
          Components: runner-direct
            Reporter: Thomas Groh
            Priority: Minor

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how
to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints
that can be resumed from can also be finalized, but otherwise the two are unrelated. This
also improves ease of never multi-finalizing a checkpoint. This gets more complicated with
retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory
handle retries properly.

This message was sent by Atlassian JIRA

View raw message