samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yan Fang" <yanfang...@gmail.com>
Subject Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store
Date Wed, 22 Apr 2015 00:00:32 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
-----------------------------------------------------------

Review request for samza.


Bugs: SAMZA-625
    https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
-------

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some jave code has
dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic mimics what
happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs
-----

  build.gradle 97de3a2 
  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaTaskConfig.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala f68a7fe 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/storage/MockCheckpointManagerFactory.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java PRE-CREATION

  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java d5e24f2 
  samza-log4j/src/test/java/org/apache/samza/config/TestLog4jSystemConfig.java 6314a3e 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
-------

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message