samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini" <criccom...@apache.org>
Subject Re: Review Request 13725: SAMZA-2
Date Thu, 03 Oct 2013 21:32:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13725/
-----------------------------------------------------------

(Updated Oct. 3, 2013, 9:32 p.m.)


Review request for samza.


Repository: samza


Description (updated)
-------

adding docs for default chooser.


adding metrics and logging.


Adding metrics to round robin chooser. updating MessageChooserFactory to take a metrics registry.


adding logs to kafka system admin


handle interrups in KafkaSystemAdmin.


rename wrappedchooser back to default chooser. fix some comments.


use topic metadata cache to get metadata


adding test for kafkasystemadmin.getlatestoffsets. bumping up gradle heap for samsa-test and
samsa-kafka tests. shutting down zookeeper in tests, so we don't leak ports.


adding a first pass at KafkaSystemAdmin.getLastOffsets


use the systemAdmins from samza container in wrapped chooser, rather than doing wiring internally.


changing ordering of wrapped chooser from bootstrap -> priority -> batching to bootstrap
-> batching -> priority.


switch default chooser to WrappedChooser. Always use wrapped chooser. Remove task.chooser.wrapped.class.


wrote default chooser test that bootstraps, prioritizes, and batches.


turning on priority tests for default chooser. refactor default chooser a bit to make it easier
to test.


test default chooser with just round robin.


bug fix to use round robin chooser factory in default chooser


change wiring for default chooser to make it more testable. add start/stop/register tests
to all choosers.


add one more test in priority chooser.


add unit test for tiered priority chooser


add unit test for bootstrapping chooser.


turn default chooser on in samza container. add license to test stateful task. add more docs
to default chooser. clean up batching unit test.


add unit test for batching chooser.


only wire in wrapping message choosers when we need them. add docs to the default chooser
factory.


rename class to BootstrappingChooser.


refactor to move into org.apache.samza.system.chooser.


build latest message offset map.


add a streams-behind-chooser that guarantees one message from each SystemStream before choose
is called.


add start/stop/register back. all tests pass.


fix bug -- should allow manual override if Int.MaxInt for bootstrap streams.


minor bug in default chooser. was re-using same chooser everywhere.


adding wiring in default chooser.


initial pass adding composed message choosers.


rebase to master, which includes SAMZA-25 metrics. fix several tests that were broken after
removing start/stop/register.


adding more docs for round robin


remove start/stop/register


cleanup some wiki markdown in MessageChooser javadoc.


add more javadocs to the message chooser.


Merge branch 'SAMZA-2_fine-grain-control-over-stream-consumption' of github.com:criccomini/incubator-samza
into SAMZA-2_fine-grain-control-over-stream-consumption


added start, stop, and register to message chooser.


adding docs for message chooser. swiching round robin chooser back to a queue.


missed license in message chooser factory


add apache licensing


samza container was using message chooser, not message chooser factory. fixed.


add stream chooser test. update stream chooser to invert priority due to bug.


add round robin test. fix compile error in round robin chooser.


add priority chooser test. fix bug in priority chooser that was reversing ordering.


adding stream chooser. adding message chooser factory.


adding priority chooser. moving default chooser to round robin chooser. adding config for
chooser


Diffs (updated)
-----

  build.gradle 077e44b0ed521504096b47f41b6180449e97280e 
  docs/learn/documentation/0.7.0/container/streams.md b15c34d3f68e32d56e0da8af91d78e47a5110f67

  samza-api/src/main/java/org/apache/samza/system/MessageChooser.java 306b2902303c72f3d7a3eb313f55d7e88d21e00d

  samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java c902d414484e05ae75c9ca58ad9629cb01120f62

  samza-api/src/main/java/org/apache/samza/system/chooser/MessageChooser.java PRE-CREATION

  samza-api/src/main/java/org/apache/samza/util/SinglePartitionSystemAdmin.java e4ed30bf363142d82a3c40909e160b5825fe60fd

  samza-core/src/main/scala/org/apache/samza/config/DefaultChooserConfig.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 0c742d83c2f60d2448a79376677713a1ff0b11ec

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 62bd243115e71612e00784124baa972b33e56cb7

  samza-core/src/main/scala/org/apache/samza/system/DefaultChooser.scala 5a72e7a3bfba0f06a5a98c6ba26865800d7780b9

  samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala d24671ec64a42ede6f779effe9c845e1cbbc5e51

  samza-core/src/main/scala/org/apache/samza/system/chooser/BatchingChooser.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/system/chooser/BootstrappingChooser.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/system/chooser/DefaultChooser.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/system/chooser/RoundRobinChooser.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/system/chooser/TieredPriorityChooser.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/chooser/MockMessageChooser.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/chooser/TestBatchingChooser.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/chooser/TestBootstrappingChooser.scala
PRE-CREATION 
  samza-core/src/test/scala/org/apache/samza/system/chooser/TestDefaultChooser.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/chooser/TestRoundRobinChooser.scala PRE-CREATION

  samza-core/src/test/scala/org/apache/samza/system/chooser/TestTieredPriorityChooser.scala
PRE-CREATION 
  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 183c6ccce39dedaef9dba56d5b61ffdedfc9d08a

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala ba08af85892f0793d01e3118fa7ad6d569a0f238

  samza-kafka/src/test/scala/org/apache/samza/checkpoint/TestKafkaCheckpointManager.scala
baf4695108b62fb2826cf0d9cdc484ba6d7bb524 
  samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala PRE-CREATION

  samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 7d4e20a8bdc7a45b0a1b464a6f4b868d1d03eab0

  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
68050528cd5f8acfe3a1f7563b4e7fe6c7473be5 

Diff: https://reviews.apache.org/r/13725/diff/


Testing
-------


Thanks,

Chris Riccomini


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message