samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dnishimura <...@git.apache.org>
Subject [GitHub] samza pull request #411: Move the changelog and checkpoint stream creation t...
Date Mon, 22 Jan 2018 19:13:08 GMT
GitHub user dnishimura opened a pull request:

    https://github.com/apache/samza/pull/411

    Move the changelog and checkpoint stream creation to the job coordinators.

    **Overview**
    The purpose of this PR is to consolidate the creation of the changelog and checkpoint
streams into the JobCoordinators. In the current state, the changelog stream is created from
the JobModelManager and the checkpoint stream is created within the OffsetManager. The issue
with creating the checkpoint in the OffsetManager is that the first call happens from the
first SamzaContainer that runs and each subsequent SamzaContainer run will attempt to create
the checkpoint stream.
    
    **Motivations**
    There are three driving forces for this refactoring. The first motivation is to assign
the creation of the changelog and checkpoint streams to the JobCoordinators where it is most
appropriate. This was discussed in more detail with @nickpan47 . The second motivation is
to have any potential failure to stream creation happen no later than during job coordination.
The third motivation is to accommodate future security work to provide a robust way to set
ACLs on streams. 
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dnishimura/samza samza-1555-move-changelog-checkpoint-creation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/411.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #411
    
----
commit 971fa91afb7f57fc8c099a7fb17e7bc96bc826c2
Author: Daniel Nishimura <dnishimura@...>
Date:   2018-01-16T21:20:07Z

    Move the responsibility of changelog and checkpoint stream creation to the job coordinators.

----


---

Mime
View raw message