samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickpan47 <...@git.apache.org>
Subject [GitHub] samza pull request #475: SAMZA-1659: Serializable OperatorSpec
Date Mon, 16 Apr 2018 04:41:32 GMT
GitHub user nickpan47 opened a pull request:

    https://github.com/apache/samza/pull/475

    SAMZA-1659: Serializable OperatorSpec

    This change is to make the user supplied functions serializable. Hence, making the full
user defined DAG serializable.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nickpan47/samza serializable-opspec-only-Jan-24-18

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #475
    
----
commit 5573a069e95f6467b92d73b3cba37f035e067ae2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-12T04:20:36Z

    WIP: new api revision

commit 8bb975204bd8a5ad3459642609c9e4992c495701
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-12T07:38:40Z

    WIP: proto-type of input/output stream/system specs

commit aeb457303779e7f31b114ad23bfa9ebaafeddab6
Author: Xinyu Liu <xiliu@...>
Date:   2017-06-29T00:16:10Z

    SAMZA-1321: Propagate end-of-stream and watermark messages
    
    The patch completes the end-of-stream work flow across multi-stage pipeline. It also contains
initial commit for supporting watermarks. For watermark, there are issues raised in the review
feedback and will be addressed by further prs. The main logic this patch adds:
    
    - EndOfStreamManager aggregates the end-of-stream control messages, propagate the result
to to downstream intermediate topics based on the topology of the IO in the StreamGraph.
    
    - WatermarkManager aggregates the watermark control messages from the upstage tasks, pass
it through the operators, and propagate it to downstream.
    
    In operator impl, I implemented similar watermark logic as Beam for watermark propagation:
    * InputWatermark(op) = min { OutputWatermark(op') | op1 is upstream of op}
    * OutputWatermark(op) = min { InputWatermark(op), OldestWorkTime(op) }
    
    Add quite a few unit tests and integration test. The code is 100% covered as reported
by Intellij. Both control messages work as expected.
    
    Author: Xinyu Liu <xiliu@xiliu-ld.linkedin.biz>
    
    Reviewers: Yi Pan <nickpan47@gmail.com>
    
    Closes #236 from xinyuiscool/SAMZA-1321

commit ae3dc6ff133fa3f0c7d9f27b6f724c652105680c
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-12T04:20:36Z

    WIP: new api revision

commit 91f364f1e3e350c4c9d6413e620d9506a1da91c8
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-12T07:38:40Z

    WIP: proto-type of input/output stream/system specs

commit b898e6c037e4fb4c9ddf1baee95095045e6500ab
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-13T01:43:50Z

    WIP: new-api-v2

commit cd528c1c30646a3307c81770f3dc82482bba1aba
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-21T16:12:41Z

    WIP: updated spec and user DAG API

commit 0bc7ee7babd0a64ed4d7cf6b7fab24be3a3447ae
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-06-30T16:55:09Z

    WIP: update the user code example on new APIs

commit 51541e133216b9c730f57653f5f7afb6c56d21a5
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-07-11T16:07:32Z

    WIP: cleanup StreamDescriptor

commit 3c50629eb93b674731398f9f8c756e499da5faf2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-07-27T22:05:07Z

    WIP: adding support for low-level task APIs

commit 4a6a58dcb80fb2919f5bad42f6f1979960826d62
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-07-31T09:51:30Z

    WIP: updated w/ low-level task API and global var ingestion/metrics reporter

commit 256155ad530c48af0cf60ba8945e8face0133a90
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-07-31T17:29:22Z

    WIP: new API user code examples

commit f227380f20503734decca75c0f34aa5810b7f1c0
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-08-02T07:50:54Z

    WIP: update the app runner classes

commit e6fb96e574aa9ee78b3beedd6921e4a680fa9c73
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-08-08T14:35:39Z

    WIP: merged all application types into StreamApplications

commit 525d8bc1b96149299ad831d1ba3d458d6dd3e4b1
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-08-09T18:02:31Z

    Merge branch '0.14.0' into new-api-v2
    
    Conflicts:
    	samza-core/src/main/java/org/apache/samza/operators/StreamGraphImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OutputOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/processor/StreamProcessor.java
    	samza-core/src/main/java/org/apache/samza/runtime/LocalContainerRunner.java

commit 6fc6d4c09d13132ae3c915de6bb6223206cb4ca2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-08-25T16:49:18Z

    WIP: experiment code to implement an end-to-end working example for new APIs

commit d7df6ed0ef3f6ca8ad5ee4cb7e7307c314a71619
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-08-31T22:41:41Z

    WIP: added all unit test for OperatorSpec#copy methods.

commit dde1ab14bee0ff4ebbc15500a938b191295dbd90
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-09-06T16:43:37Z

    WIP: first end-to-end test

commit 50201728e964cbe3c997b4e335df3cd2a46ed076
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-10-19T23:48:56Z

    Merge branch 'experiment-new-api-v2' into new-api-v2-0.14
    
    Conflicts:
    	samza-api/src/main/java/org/apache/samza/operators/MessageStream.java
    	samza-api/src/main/java/org/apache/samza/operators/StreamGraph.java
    	samza-api/src/main/java/org/apache/samza/operators/windows/Windows.java
    	samza-api/src/main/java/org/apache/samza/operators/windows/internal/WindowInternal.java
    	samza-api/src/main/java/org/apache/samza/serializers/Serde.java
    	samza-api/src/main/java/org/apache/samza/system/StreamSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/MessageStreamImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/StreamGraphImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/functions/PartialJoinFunction.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/InputOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OperatorImplGraph.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OutputOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/InputOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/JoinOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpecs.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OutputOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OutputStreamImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/SinkOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/StreamOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/WindowOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/runtime/AbstractApplicationRunner.java
    	samza-core/src/main/java/org/apache/samza/runtime/LocalApplicationRunner.java
    	samza-core/src/main/java/org/apache/samza/runtime/RemoteApplicationRunner.java
    	samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
    	samza-core/src/test/java/org/apache/samza/control/TestEndOfStreamManager.java
    	samza-core/src/test/java/org/apache/samza/control/TestIOGraph.java
    	samza-core/src/test/java/org/apache/samza/control/TestWatermarkManager.java
    	samza-core/src/test/java/org/apache/samza/example/BroadcastExample.java
    	samza-core/src/test/java/org/apache/samza/example/MergeExample.java
    	samza-core/src/test/java/org/apache/samza/execution/TestExecutionPlanner.java
    	samza-core/src/test/java/org/apache/samza/execution/TestJobGraphJsonGenerator.java
    	samza-core/src/test/java/org/apache/samza/operators/TestJoinOperator.java
    	samza-core/src/test/java/org/apache/samza/operators/TestMessageStreamImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/TestStreamGraphImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/TestWindowOperator.java
    	samza-core/src/test/java/org/apache/samza/operators/impl/TestOperatorImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/impl/TestOperatorImplGraph.java
    	samza-core/src/test/java/org/apache/samza/operators/spec/TestWindowOperatorSpec.java
    	samza-core/src/test/java/org/apache/samza/runtime/TestApplicationRunnerMain.java
    	samza-core/src/test/java/org/apache/samza/runtime/TestLocalApplicationRunner.java
    	samza-test/src/main/java/org/apache/samza/example/KeyValueStoreExample.java
    	samza-test/src/main/java/org/apache/samza/example/OrderShipmentJoinExample.java
    	samza-test/src/main/java/org/apache/samza/example/PageViewCounterExample.java
    	samza-test/src/main/java/org/apache/samza/example/RepartitionExample.java
    	samza-test/src/main/java/org/apache/samza/example/WindowExample.java
    	samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
    	samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
    	samza-test/src/test/java/org/apache/samza/test/operator/RepartitionWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/SessionWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/StreamApplicationIntegrationTestHarness.java
    	samza-test/src/test/java/org/apache/samza/test/operator/TestRepartitionWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/TumblingWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/processor/TestZkLocalApplicationRunner.java

commit bf1ce907645e4a131c5765c4612c158e1855730a
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-10-20T00:43:33Z

    WIP: removing StreamDescriptor first

commit d46403294aecc2ebdbc3a890da8c40acfa35f448
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-10-23T16:05:48Z

    WIP: fixing unit tests after merge

commit 6a14b2afad8d57cccb17e5926cd44743b6ebd034
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-10-30T22:25:03Z

    WIP: fixed unit test failure for Windows

commit 475a46bced8d84c7fe09b57590d3a72e0587c3c8
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-12-15T18:59:29Z

    WIP: fixed TestZkLocalApplicationRunner. Debugging issues w/ TestRepartitionWindowApp
(i.e. missing changelog creation step when directly running LocalApplicationRunner)

commit dc7da87e2eec618bb060acfec695c5b1aa280259
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2017-12-22T18:27:21Z

    WIP: unit tests for serialization

commit 4102aa8ced2b5630edf3ac84e0c6298956a999b5
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-01-02T17:36:29Z

    WIP: continued working on potential offspring integration

commit 1670aff0cefb569b109fad3928f04190a7b4e0a1
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-01-10T22:20:22Z

    WIP: class-loading of user program logic and main() method based user program logic are
both included in ThreadJobFactory/ProcessJobFactory/YarnJobFactory. ThreadJobFactory test
suite to be fixed.

commit 0ebebfc3ba49de6d6589cd3b82eb8ce3882d034d
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-01-19T23:11:26Z

    WIP: serialization only change

commit aca423085ed29452adc89c48aee4aa1126874328
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-01-25T07:52:16Z

    WIP: Serialize OperatorSpec only w/o StreamApplication interface change. Passed all build
and tests.

commit b973b105ddb8cd49ea5f084ceebea19b27254f36
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-02-08T08:12:58Z

    Merge branch 'master' into serializable-opspec-only-Jan-24-18
    
    Conflicts:
    	samza-api/src/main/java/org/apache/samza/operators/MessageStream.java
    	samza-core/src/main/java/org/apache/samza/operators/MessageStreamImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/StreamGraphImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OperatorImplGraph.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/OutputOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/PartialJoinOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/PartitionByOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/impl/WindowOperatorImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/InputOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OperatorSpecs.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/OutputStreamImpl.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/PartitionByOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/spec/StreamOperatorSpec.java
    	samza-core/src/main/java/org/apache/samza/operators/stream/IntermediateMessageStreamImpl.java
    	samza-core/src/main/java/org/apache/samza/runtime/LocalApplicationRunner.java
    	samza-core/src/main/scala/org/apache/samza/coordinator/JobModelManager.scala
    	samza-core/src/test/java/org/apache/samza/execution/TestExecutionPlanner.java
    	samza-core/src/test/java/org/apache/samza/execution/TestJobGraphJsonGenerator.java
    	samza-core/src/test/java/org/apache/samza/operators/TestJoinOperator.java
    	samza-core/src/test/java/org/apache/samza/operators/TestMessageStreamImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/TestStreamGraphImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/impl/TestOperatorImpl.java
    	samza-core/src/test/java/org/apache/samza/operators/impl/TestOperatorImplGraph.java
    	samza-core/src/test/java/org/apache/samza/operators/impl/TestWindowOperator.java
    	samza-core/src/test/java/org/apache/samza/operators/spec/TestWindowOperatorSpec.java
    	samza-test/src/main/java/org/apache/samza/example/KeyValueStoreExample.java
    	samza-test/src/main/java/org/apache/samza/example/OrderShipmentJoinExample.java
    	samza-test/src/main/java/org/apache/samza/example/PageViewCounterExample.java
    	samza-test/src/main/java/org/apache/samza/example/RepartitionExample.java
    	samza-test/src/main/java/org/apache/samza/example/WindowExample.java
    	samza-test/src/test/java/org/apache/samza/test/controlmessages/EndOfStreamIntegrationTest.java
    	samza-test/src/test/java/org/apache/samza/test/controlmessages/WatermarkIntegrationTest.java
    	samza-test/src/test/java/org/apache/samza/test/operator/RepartitionJoinWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/SessionWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/TestRepartitionJoinWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/operator/TumblingWindowApp.java
    	samza-test/src/test/java/org/apache/samza/test/processor/TestZkLocalApplicationRunner.java

commit 7c8d1591cef6326f32b2ffd061cc52b902e37214
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date:   2018-02-12T08:29:26Z

    WIP: working on unit tests for trigger, broadcast, join, table, and SQL UDF function serialization

----


---

Mime
View raw message