samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naveen Somasundaram" <navg...@gmail.com>
Subject Re: Review Request 31718: Samza 465 [Draft]
Date Mon, 16 Mar 2015 21:48:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31718/#review76603
-----------------------------------------------------------



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/31718/#comment124218>

    removedf



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/31718/#comment124219>

    removed



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/31718/#comment124221>

    It was a left over from my first version, removed



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/31718/#comment124222>

    removed



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/31718/#comment124247>

    There was a double start issue happening on the coordinator stream consumers because of
this, but that's fixed now. I'm adding start again.



samza-api/src/main/java/org/apache/samza/container/TaskName.java
<https://reviews.apache.org/r/31718/#comment124170>

    moved to javadocs



samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java
<https://reviews.apache.org/r/31718/#comment124171>

    Removed because this adds dependency on core from api (because of java doc)



samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java
<https://reviews.apache.org/r/31718/#comment124172>

    done



samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java
<https://reviews.apache.org/r/31718/#comment124206>

    done



samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java
<https://reviews.apache.org/r/31718/#comment124207>

    Looks like it can be private



samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java
<https://reviews.apache.org/r/31718/#comment124209>

    added



samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java
<https://reviews.apache.org/r/31718/#comment124210>

    fixed



samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java
<https://reviews.apache.org/r/31718/#comment124211>

    done



samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java
<https://reviews.apache.org/r/31718/#comment124212>

    Yes



samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java
<https://reviews.apache.org/r/31718/#comment124213>

    yes



samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java
<https://reviews.apache.org/r/31718/#comment124214>

    fixed



samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala
<https://reviews.apache.org/r/31718/#comment124215>

    done



samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/31718/#comment124249>

    removed


- Naveen Somasundaram


On March 4, 2015, 1:39 a.m., Naveen Somasundaram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31718/
> -----------------------------------------------------------
> 
> (Updated March 4, 2015, 1:39 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Samza 465 [Draft]
> 
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single
entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager
will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from
jobcoordinator.
> 
> Pending Issues:
> Please note that this is a draft patch (unit tests pass, but still need to run Zopkio
integration test) and do some cosemtic checks and add more docs after testing.
> 1. The failure scenario (when the container fails and the jobcoordinator serving the
latest offsets) has still not been addressed, I am working on it now. 
> 2. The metrics registry is right now not used correctly, I need to pass the right reference
to it
> 3. The coordinator stream does not use the same consumer and producer from the "systemconsumers"
and "systemproducers" we have.
> 4. Write unit tests to test Changlogmanager's write and read.
> 5. Write/extend test to verify when no offsets are passed for correct behavior.
> 
> 
> Diffs
> -----
> 
>   build.gradle 38383bd9e3f0847d6088a4ea4c1ee6f3dcd1e430 
>   samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java 593d11872430100e000c7d4b6edc5ef29649d8d4

>   samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5

>   samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d

>   samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f

>   samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998

>   samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d

>   samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java
01997ae22641b735cd452a0e89a49219e2874892 
>   samza-core/src/main/java/org/apache/samza/changelog/ChangelogManager.java PRE-CREATION

>   samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc

>   samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java
7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde 
>   samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java
3517912eaafbf95f8c8cc70ab5869548a56b76e7 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93

>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0

>   samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala
2a87a6e0cef72179b5383fc824266de1f9606293 
>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9

>   samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02

>   samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e

>   samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca

>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 8a6d8656c14ad9c7cc7b5d9a39f1f733afd71a88

>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545

>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 16345cd1c1354a0d25a0000d81a307dbe3abbe81

>   samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 6985af6e7cc0d408fa07fbac60141d1126323777

>   samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010

>   samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala
dcf0435351cb5deffc7dc4a0cd438013939d54c0 
>   samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8

>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fee24614fce101e91c4f933d9b4e65dda0a

>   samza-core/src/main/scala/org/apache/samza/system/chooser/DefaultChooser.scala 95bd18898dd9e2b6848523fe89c9017a7267ab3b

>   samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala
ec1d749cb5186f788b402877996a4caa37e99362 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca

>   samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java
76bc6810a3162a1dc58a36033b3b1f75616bd6ca 
>   samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64

>   samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf

>   samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d

>   samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala
10ff1f437220b38810f8a32903cc72df20f206d3 
>   samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala acded7d5e19271edcf21f8e77c74bac4f376ea3a

>   samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala
1ee5c06fbd1be5e4ce944a16454c8bd32459d395 
>   samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289

>   samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala
9a8e1450a6cc14713817f719cfa56a0e5c97a6f6 
>   samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4

>   samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala b75f44060fb8e660e824eaeb9cfdcc9d6fa902e8

>   samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala
5d8ee4fa74be9ba36956c11ae33573be2d2d5826 
>   samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
1d5627d0c561a0be6b48ee307b755958e62b783e 
>   samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
f2defbd39708e959edb1d6674e542b5bc9e02666 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala e57b8ba1e09765774314ec469645b5d0bbde060f

>   samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9

>   samza-kafka/src/main/scala/org/apache/samza/serializers/KafkaSerde.scala 82ba2a09b98a04ac64301743b3ae32f29cefbc3b

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala b790be17cfe08da28220ffb381cbd618ebe25cf0

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4506ea367eec4e40da45feee777ba73069025a4c

>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
a0e1ccbfe9dc4fd26ca6b30fc2d1348fb7d007e4 
>   samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala
b76d5ad68640908bef552125d405b467386025f8 
>   samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
553d6b4d6ffe21f4a92c8c347e835d95d71b5863 
>   samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala
c759a7bea7b67714eaa90a97f828079f26acbca4 
>   samza-kv/src/main/scala/org/apache/samza/storage/kv/NullSafeKeyValueStore.scala 4f48cf490d6c1012591a602c0d29dcc71473090f

>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 5f5195c8f303261cde069390db543ee86d43148d

>   samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 9a9d6483ffb92ba6ad5c730deec87651a7cbc64a

>   samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb

>   samza-test/src/main/scala/org/apache/samza/test/performance/TestKeyValuePerformance.scala
b4fa7d3575e3b81ea3b606a1de8f06cedb87168a 
>   samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala
45c76e86361b9e1a54cb5fc717126d36b64cf7e8 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala
ca25258217e5ebc44b34fbc4d69ecb28c81df618 
>   samza-test/src/test/scala/org/apache/samza/test/performance/TestSamzaContainerPerformance.scala
d589d762a18f9425aa8d8dd589011a151bcb59a4 
>   samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215

>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2

>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterMetrics.scala ee2aa32fdfaf439f669c51afd91f2390e0b3fd09

>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d

>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala
b0b6543856cb87888c5a719182ad9576b51bba1a 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5

>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala
765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4 
>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala
81dea9d6d1921462b200c62dbdf016c0eb2f01b2 
>   samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala
58f2464211a1fb7ff40f5978fd41f64d088002d0 
> 
> Diff: https://reviews.apache.org/r/31718/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Naveen Somasundaram
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message