samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shadi A. Noghabi" <abdol...@illinois.edu>
Subject Re: Review Request 36471: Autoscaling for samza (work in progress)
Date Fri, 14 Aug 2015 18:21:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36471/
-----------------------------------------------------------

(Updated Aug. 14, 2015, 6:20 p.m.)


Review request for samza and Navina Ramesh.


Changes
-------

Separated config for auto-scaling, and added stat collection into csv files.


Repository: samza


Description
-------

This work is for SAMZA-719. Currently, a fixed number of containers is assigned to a job as
an input configuration parameter. However, with this design jobs can fail due to lack of enough
resources (such as memory), or they can become a bottleneck in a workflow containing many
jobs. While auto-scaling is much broader term, the goal of this project will be to enable
a Samza job to automatically scale its containers such that there is improved job performance.

Based on the design, we need a profiler, analyzer, optimizer and deployer module.

-currently, all components added based on memory usage and the loop seems to work in inital
test

-tests not added for all components, and further testing is needed.


Diffs (updated)
-----

  build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 
  checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c 
  docs/learn/documentation/versioned/jobs/configuration-table.html ea73b40a054002a8f4b0e1fca9e271aae59148ab

  gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 
  samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingConfig.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingMode.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingSystem.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/Analyzer.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/MemoryAnalyzer.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/deployer/Deployer.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/deployer/RestartingJobDeployer.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MemoryMetrics.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MetricsHistory.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/metrics/SlidingWindowMetric.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/metrics/StatCollector.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModel.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/Optimizer.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizer.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerOptimizer.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/autoScaling/profiler/Profiler.java PRE-CREATION

  samza-core/src/main/java/org/apache/samza/autoScaling/profiler/SnapshotReporterProfiler.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemConsumer.java
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc 
  samza-core/src/main/scala/org/apache/samza/autoScaling/AutoScalingCommandLine.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemFactory.scala
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/config/AutoScalingConfigRewriter.scala PRE-CREATION

  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4da6649eb78753ba3b3f529373b6f2dbe4

  samza-core/src/main/scala/org/apache/samza/config/MetricsConfig.scala f7a11c575807c9d47e441bb8c6b0616142700d37

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517048ad5730762506426ee7578c66181db8

  samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByContainerCountFactory.scala
eca6215a1b84c6dc0a4aa1e9e0f8d9f8bff1b467 
  samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByJobModel.scala
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByjobModelFactory.scala
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 0dbf14b8bbb8236a8cdbe037a99dec91c5e63965

  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2

  samza-core/src/main/scala/org/apache/samza/metrics/JvmMetrics.scala 64daa0f0bb7c6350a57e06705aef41b781779416

  samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporter.scala
b6696f823e11a0c2134fc830178e47c3ee857378 
  samza-core/src/main/scala/org/apache/samza/metrics/reporter/MetricsSnapshotReporterFactory.scala
dcf0435351cb5deffc7dc4a0cd438013939d54c0 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 419452c0d48d8faa84279fcf02a03e87309821d8

  samza-core/src/test/java/org/apache/samza/autoScaling/AutoScalingConfigTest.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/autoScaling/SlidingWindowMetricTest.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/autoScaling/analyzer/MockAnalyzer.java PRE-CREATION

  samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModelTest.java
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/MemoryOptimizerTest.java
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizerTest.java
PRE-CREATION 
  samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByJobModel.scala
PRE-CREATION 
  samza-shell/src/main/bash/run-auto-scaling.sh PRE-CREATION 
  samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION 
  samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ea702a919348305ff95ce0b4ca1996a13aff04ec

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala af42c6a6636953a95f79837fe372e0dbd735df70

  samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala 7b7d86a43c69e72c47eaa91f68be24e0f4022891


Diff: https://reviews.apache.org/r/36471/diff/


Testing
-------

tested with hello samza


Thanks,

Shadi A. Noghabi


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message