samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ahmedahamid <...@git.apache.org>
Subject [GitHub] samza pull request #637: SAMZA-1860: Modularize Join input validation in Exe...
Date Wed, 12 Sep 2018 05:27:20 GMT
GitHub user ahmedahamid opened a pull request:

    https://github.com/apache/samza/pull/637

    SAMZA-1860: Modularize Join input validation in ExecutionPlanner

    This change breaks down the validation of partition counts of input and
    intermediate streams participating in Join operations into 3 separate steps:
    1. Grouping `InputOperatorSpec`s by the `JoinOperatorSpec`s of the Join operations they
participate in
    2. Replacing `InputOperatorSpec`s with their corresponding `StreamEdge`s
    3. Verifying/Inferring partition counts of input/intermediate streams

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ahmedahamid/samza dev/ahabdulh/modularize-exec-planner

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #637
    
----
commit 88fdad8c004243b376b1980999ed82581d3be796
Author: Ahmed Abdul Hamid <ahabdulh@...>
Date:   2018-09-05T02:02:11Z

    SAMZA-1838: Make some minor improvements to ExecutionPlanner
    
    This commit includes the following changes:
      - Fix case where ExecutionPlanner did not throw in response
        to joining 2 input streams with different partition counts
      - Improve some method names in ExecutionPlanner
      - Improve some method/field names in JobGraph
      - Make minor improvements to createJobGraph()
      - Rewrite updateExistingPartitions() to make it a little easier to follow
      - Use more constrained OperatorSpec types in the associations defined in
        calculateJoinInputPartitions()
      - Have calculateIntStreamPartitions() throw in response to bad config for
        job.intermediate.stream.partitions
      - Improve some error messages

commit c8763d2ed60e376cd81734c6461817fa7c9fd3f9
Author: Ahmed Abdul Hamid <ahabdulh@...>
Date:   2018-09-12T04:42:13Z

    SAMZA-1860: Modularize Join input validation in ExecutionPlanner
    
    This change breaks down the validation of partition counts of input and
    intermediate streams participating in Join operations into 3 separate steps:
        1. Grouping InputOperatorSpecs by the JoinOperatorSpecs of the Join
           operations they participate in
        2. Replacing InputOperatorSpecs with their corresponding StreamEdges
        3. Verifying/Inferring partition counts of input/intermediate streams

----


---

Mime
View raw message