samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sriram Subramanian" <sriram....@gmail.com>
Subject Re: Review Request 13725: SAMZA-2
Date Fri, 04 Oct 2013 20:56:58 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13725/#review26699
-----------------------------------------------------------

Ship it!


Ship It!

- Sriram Subramanian


On Oct. 3, 2013, 9:32 p.m., Chris Riccomini wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13725/
> -----------------------------------------------------------
> 
> (Updated Oct. 3, 2013, 9:32 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> adding docs for default chooser.
> 
> 
> adding metrics and logging.
> 
> 
> Adding metrics to round robin chooser. updating MessageChooserFactory to take a metrics
registry.
> 
> 
> adding logs to kafka system admin
> 
> 
> handle interrups in KafkaSystemAdmin.
> 
> 
> rename wrappedchooser back to default chooser. fix some comments.
> 
> 
> use topic metadata cache to get metadata
> 
> 
> adding test for kafkasystemadmin.getlatestoffsets. bumping up gradle heap for samsa-test
and samsa-kafka tests. shutting down zookeeper in tests, so we don't leak ports.
> 
> 
> adding a first pass at KafkaSystemAdmin.getLastOffsets
> 
> 
> use the systemAdmins from samza container in wrapped chooser, rather than doing wiring
internally.
> 
> 
> changing ordering of wrapped chooser from bootstrap -> priority -> batching to
bootstrap -> batching -> priority.
> 
> 
> switch default chooser to WrappedChooser. Always use wrapped chooser. Remove task.chooser.wrapped.class.
> 
> 
> wrote default chooser test that bootstraps, prioritizes, and batches.
> 
> 
> turning on priority tests for default chooser. refactor default chooser a bit to make
it easier to test.
> 
> 
> test default chooser with just round robin.
> 
> 
> bug fix to use round robin chooser factory in default chooser
> 
> 
> change wiring for default chooser to make it more testable. add start/stop/register tests
to all choosers.
> 
> 
> add one more test in priority chooser.
> 
> 
> add unit test for tiered priority chooser
> 
> 
> add unit test for bootstrapping chooser.
> 
> 
> turn default chooser on in samza container. add license to test stateful task. add more
docs to default chooser. clean up batching unit test.
> 
> 
> add unit test for batching chooser.
> 
> 
> only wire in wrapping message choosers when we need them. add docs to the default chooser
factory.
> 
> 
> rename class to BootstrappingChooser.
> 
> 
> refactor to move into org.apache.samza.system.chooser.
> 
> 
> build latest message offset map.
> 
> 
> add a streams-behind-chooser that guarantees one message from each SystemStream before
choose is called.
> 
> 
> add start/stop/register back. all tests pass.
> 
> 
> fix bug -- should allow manual override if Int.MaxInt for bootstrap streams.
> 
> 
> minor bug in default chooser. was re-using same chooser everywhere.
> 
> 
> adding wiring in default chooser.
> 
> 
> initial pass adding composed message choosers.
> 
> 
> rebase to master, which includes SAMZA-25 metrics. fix several tests that were broken
after removing start/stop/register.
> 
> 
> adding more docs for round robin
> 
> 
> remove start/stop/register
> 
> 
> cleanup some wiki markdown in MessageChooser javadoc.
> 
> 
> add more javadocs to the message chooser.
> 
> 
> Merge branch 'SAMZA-2_fine-grain-control-over-stream-consumption' of github.com:criccomini/incubator-samza
into SAMZA-2_fine-grain-control-over-stream-consumption
> 
> 
> added start, stop, and register to message chooser.
> 
> 
> adding docs for message chooser. swiching round robin chooser back to a queue.
> 
> 
> missed license in message chooser factory
> 
> 
> add apache licensing
> 
> 
> samza container was using message chooser, not message chooser factory. fixed.
> 
> 
> add stream chooser test. update stream chooser to invert priority due to bug.
> 
> 
> add round robin test. fix compile error in round robin chooser.
> 
> 
> add priority chooser test. fix bug in priority chooser that was reversing ordering.
> 
> 
> adding stream chooser. adding message chooser factory.
> 
> 
> adding priority chooser. moving default chooser to round robin chooser. adding config
for chooser
> 
> 
> Diffs
> -----
> 
>   build.gradle 077e44b0ed521504096b47f41b6180449e97280e 
>   docs/learn/documentation/0.7.0/container/streams.md b15c34d3f68e32d56e0da8af91d78e47a5110f67

>   samza-api/src/main/java/org/apache/samza/system/MessageChooser.java 306b2902303c72f3d7a3eb313f55d7e88d21e00d

>   samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java c902d414484e05ae75c9ca58ad9629cb01120f62

>   samza-api/src/main/java/org/apache/samza/system/chooser/MessageChooser.java PRE-CREATION

>   samza-api/src/main/java/org/apache/samza/util/SinglePartitionSystemAdmin.java e4ed30bf363142d82a3c40909e160b5825fe60fd

>   samza-core/src/main/scala/org/apache/samza/config/DefaultChooserConfig.scala PRE-CREATION

>   samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 0c742d83c2f60d2448a79376677713a1ff0b11ec

>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 62bd243115e71612e00784124baa972b33e56cb7

>   samza-core/src/main/scala/org/apache/samza/system/DefaultChooser.scala 5a72e7a3bfba0f06a5a98c6ba26865800d7780b9

>   samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala d24671ec64a42ede6f779effe9c845e1cbbc5e51

>   samza-core/src/main/scala/org/apache/samza/system/chooser/BatchingChooser.scala PRE-CREATION

>   samza-core/src/main/scala/org/apache/samza/system/chooser/BootstrappingChooser.scala
PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/system/chooser/DefaultChooser.scala PRE-CREATION

>   samza-core/src/main/scala/org/apache/samza/system/chooser/RoundRobinChooser.scala PRE-CREATION

>   samza-core/src/main/scala/org/apache/samza/system/chooser/TieredPriorityChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala PRE-CREATION

>   samza-core/src/test/scala/org/apache/samza/system/chooser/MockMessageChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/chooser/TestBatchingChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/chooser/TestBootstrappingChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/chooser/TestDefaultChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/chooser/TestRoundRobinChooser.scala
PRE-CREATION 
>   samza-core/src/test/scala/org/apache/samza/system/chooser/TestTieredPriorityChooser.scala
PRE-CREATION 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 183c6ccce39dedaef9dba56d5b61ffdedfc9d08a

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala ba08af85892f0793d01e3118fa7ad6d569a0f238

>   samza-kafka/src/test/scala/org/apache/samza/checkpoint/TestKafkaCheckpointManager.scala
baf4695108b62fb2826cf0d9cdc484ba6d7bb524 
>   samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala
PRE-CREATION 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala
7d4e20a8bdc7a45b0a1b464a6f4b868d1d03eab0 
>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
68050528cd5f8acfe3a1f7563b4e7fe6c7473be5 
> 
> Diff: https://reviews.apache.org/r/13725/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Chris Riccomini
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message