samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject Re: Review Request 44920: SAMZA-881
Date Thu, 24 Mar 2016 21:41:59 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44920/
-----------------------------------------------------------

(Updated March 24, 2016, 9:41 p.m.)


Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Yi Pan (Data Infrastructure),
Navina Ramesh, and Xinyu Liu.


Summary (updated)
-----------------

SAMZA-881


Repository: samza


Description (updated)
-------

1.Proposed new APIs for running Samza without Yarn. (SAMZA-881)
   - Defined the ContainerProcessManager abstraction.
   - Defined the SamzaResource, SamzaResourceRequest. 
   - Re-wrote the SamzaAppMaster logic into a ClusterBasedJobCoordinator.
2.Defined a ClusterManagerConfig to handle configurations independent of Yarn/Mesos.
3.Made Samza completely independent of Yarn. This cleanly separates Samza specific components
and Yarn
specific components.
4.Readability improvements to the existing code base.
   -Added explicity documentation for every method, member and class (including on thread-safety)
   - Made internal variables final to document intent, visibility across threads. (trivially
by adding modifiers, or by changing where they're initialized.)
5.Refactored JobCoordinator into a JobModelReader.

== Diff2 ==
Address review feedback.


Diffs (updated)
-----

  samza-core/src/main/java/org/apache/samza/clustermanager/AbstractContainerAllocator.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/clustermanager/ClusterBasedJobCoordinator.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/clustermanager/ContainerAllocator.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManagerFactory.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/ContainerProcessManager.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/ContainerRequestState.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/HostAwareContainerAllocator.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/clustermanager/ResourceFailure.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaAppState.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaContainerLaunchException.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaResource.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaResourceRequest.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaResourceStatus.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/clustermanager/SamzaTaskManager.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/config/ClusterManagerConfig.java PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/coordinator/JobModelReader.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/metrics/SamzaAppMasterMetrics.scala PRE-CREATION

  samza-shell/src/main/bash/run-am.sh 9545a96953baaff17ad14962e02bc12aadbb1101 
  samza-yarn/src/main/java/org/apache/samza/job/yarn/refactor/YarnAppState.java PRE-CREATION

  samza-yarn/src/main/java/org/apache/samza/job/yarn/refactor/YarnContainerManager.java PRE-CREATION

  samza-yarn/src/main/java/org/apache/samza/job/yarn/refactor/YarnContainerManagerFactory.java
PRE-CREATION 
  samza-yarn/src/main/java/org/apache/samza/job/yarn/refactor/YarnContainerRunner.java PRE-CREATION

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/refactor/SamzaAppMasterService.scala
PRE-CREATION 
  samza-yarn/src/main/scala/org/apache/samza/job/yarn/refactor/SamzaYarnAppMasterLifecycle.scala
PRE-CREATION 

Diff: https://reviews.apache.org/r/44920/diff/


Testing (updated)
-------

Tested with sample jobs on clusters of varying sizes. Tested locally. TODO: Unit tests.


Thanks,

Jagadish Venkatraman


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message