samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yan Fang" <yanfang...@gmail.com>
Subject Re: Review Request 34974: SAMZA-676: implement broadcast stream
Date Wed, 29 Jul 2015 22:49:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34974/
-----------------------------------------------------------

(Updated July 29, 2015, 10:49 p.m.)


Review request for samza.


Changes
-------

remove whiltespaces
update to latest master


Bugs: SAMZA-676
    https://issues.apache.org/jira/browse/SAMZA-676


Repository: samza


Description
-------

1. added offsetComparator method in SystemAdmin Interface

2. added "task.global.inputs" config

3. rewrote Grouper classes using Java; allows to assign global streams during grouping

4. used LinkedHashSet instead of HashSet in CoordinatorStreamSystemConsumer to preserve messages
order

5. added taskNames to the offsets in OffsetManager

6. allowed to assign one SSP to multiple taskInstances

7. skipped already-processed messages in RunLoop

8. unit tests for all changes


Diffs (updated)
-----

  checkstyle/import-control.xml 6654319 
  docs/learn/documentation/versioned/container/samza-container.md 9f46414 
  docs/learn/documentation/versioned/jobs/configuration-table.html ea73b40 
  samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 7a588eb 
  samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
249b8ae 
  samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupByPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartitionFactory.java
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 20e5d26 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4 
  samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala c292ae4 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517 
  samza-core/src/main/scala/org/apache/samza/container/TaskInstance.scala c5a5ea5 
  samza-core/src/main/scala/org/apache/samza/container/TaskInstanceMetrics.scala 9dc7051 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupByPartition.scala
44e95fc 
  samza-core/src/main/scala/org/apache/samza/container/grouper/stream/GroupBySystemStreamPartition.scala
3c0acad 
  samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
097f410 
  samza-core/src/test/java/org/apache/samza/config/TestTaskConfigJava.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupByPartition.java
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.java
PRE-CREATION 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 8d54c46 
  samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala 64a5844 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 84fdeaa 
  samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 7caad28 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/GroupByTestBase.scala
a14169b 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupByPartition.scala
74daf72 
  samza-core/src/test/scala/org/apache/samza/container/grouper/stream/TestGroupBySystemStreamPartition.scala
deb3895 
  samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala 4097ac7

  samza-elasticsearch/src/main/java/org/apache/samza/system/elasticsearch/ElasticsearchSystemAdmin.java
1fd5dd3 
  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala 35086f5

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala de00320

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 1629035

  samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemConsumer.scala 2a84328

  samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java b063366 
  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
1e936b4 

Diff: https://reviews.apache.org/r/34974/diff/


Testing
-------


Thanks,

Yan Fang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message