samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naveen Somasundaram" <navg...@gmail.com>
Subject Re: Review Request 31718: Samza 465 [Draft]
Date Wed, 04 Mar 2015 01:25:27 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31718/
-----------------------------------------------------------

(Updated March 4, 2015, 1:25 a.m.)


Review request for samza.


Repository: samza


Description (updated)
-------

Samza 465 [Draft]

The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity
that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager
will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.

Pending Issues:
Please note that this is a draft patch (unit tests pass, but still need to run Zopkio integration
test) and do some cosemtic checks and add more docs after testing.
1. The failure scenario (when the container fails and the jobcoordinator serving the latest
offsets) has still not been addressed, I am working on it now. 
2. The metrics registry is right now not used correctly, I need to pass the right reference
to it
3. The coordinator stream does not use the same consumer and producer from the "systemconsumers"
and "systemproducers" we have.


Diffs (updated)
-----

  build.gradle 38383bd9e3f0847d6088a4ea4c1ee6f3dcd1e430 
  samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java 593d11872430100e000c7d4b6edc5ef29649d8d4

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d

  samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f

  samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998

  samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d

  samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
01997ae22641b735cd452a0e89a49219e2874892 
  samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc

  samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde

  samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7

  samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93

  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0

  samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala
2a87a6e0cef72179b5383fc824266de1f9606293 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9

  samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02

  samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e

  samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 8a6d8656c14ad9c7cc7b5d9a39f1f733afd71a88

  samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545

  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 16345cd1c1354a0d25a0000d81a307dbe3abbe81

  samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 6985af6e7cc0d408fa07fbac60141d1126323777

  samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010

  samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala
dcf0435351cb5deffc7dc4a0cd438013939d54c0 
  samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8

  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a

  samza-core/src/main/scala/org/apache/samza/system/chooser/DefaultChooser.scala 95bd18898dd9e2b6848523fe89c9017a7267ab3b

  samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
ec1d749cb5186f788b402877996a4caa37e99362 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca

  samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca

  samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf

  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d

  samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala
10ff1f437220b38810f8a32903cc72df20f206d3 
  samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala acded7d5e19271edcf21f8e77c74bac4f376ea3a

  samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala
1ee5c06fbd1be5e4ce944a16454c8bd32459d395 
  samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289

  samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6

  samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4

  samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8

  samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala
5d8ee4fa74be9ba36956c11ae33573be2d2d5826 
  samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
1d5627d0c561a0be6b48ee307b755958e62b783e 
  samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
f2defbd39708e959edb1d6674e542b5bc9e02666 
  samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala e57b8ba1e09765774314ec469645b5d0bbde060f

  samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9

  samza-kafka/src/main/scala/org/apache/samza/serializers/KafkaSerde.scala 82ba2a09b98a04ac64301743b3ae32f29cefbc3b

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala b790be17cfe08da28220ffb381cbd618ebe25cf0

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4506ea367eec4e40da45feee777ba73069025a4c

  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala a0e1ccbfe9dc4fd26ca6b30fc2d1348fb7d007e4

  samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala
b76d5ad68640908bef552125d405b467386025f8 
  samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
553d6b4d6ffe21f4a92c8c347e835d95d71b5863 
  samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala c759a7bea7b67714eaa90a97f828079f26acbca4

  samza-kv/src/main/scala/org/apache/samza/storage/kv/NullSafeKeyValueStore.scala 4f48cf490d6c1012591a602c0d29dcc71473090f

  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 5f5195c8f303261cde069390db543ee86d43148d

  samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 9a9d6483ffb92ba6ad5c730deec87651a7cbc64a

  samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb

  samza-test/src/main/scala/org/apache/samza/test/performance/TestKeyValuePerformance.scala
b4fa7d3575e3b81ea3b606a1de8f06cedb87168a 
  samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8

  samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ca25258217e5ebc44b34fbc4d69ecb28c81df618

  samza-test/src/test/scala/org/apache/samza/test/performance/TestSamzaContainerPerformance.scala
d589d762a18f9425aa8d8dd589011a151bcb59a4 
  samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterMetrics.scala ee2aa32fdfaf439f669c51afd91f2390e0b3fd09

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5

  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4

  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2

  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
58f2464211a1fb7ff40f5978fd41f64d088002d0 

Diff: https://reviews.apache.org/r/31718/diff/


Testing
-------


Thanks,

Naveen Somasundaram


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message