samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shadi A. Noghabi" <abdol...@illinois.edu>
Subject Re: Review Request 36006: Writing a tool to read from the coordinator stream and react to config changes accordingly.
Date Tue, 28 Jul 2015 21:09:18 GMT


> On July 22, 2015, 6:25 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala, line 130
> > <https://reviews.apache.org/r/36006/diff/2/?file=1015156#file1015156line130>
> >
> >     Question: can we move this code into the JobCoordinator.start()? I don't see
any reason why this has to be outside JobCoordinator?

It cannot be moved to there since the server url is not set yet in JobCoordinator.start().
I have moved this to SamzaAppMasterService.onInit()


On July 22, 2015, 6:25 p.m., Shadi A. Noghabi wrote:
> > Some high-level comments: I don't see a reason that these tools classes have to
be in samza-core. Can we move it to a separate module, like samza-man? That will also allow
us to keep the minimum dependencies in samza-core. I will continue w/ the implementation of
ConfigManager.

I moved the config manager to a separate module. What do you mean by "I will continue w/ the
implementation of ConfigManager" ?


- Shadi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36006/#review92616
-----------------------------------------------------------


On July 28, 2015, 2:39 a.m., Shadi A. Noghabi wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36006/
> -----------------------------------------------------------
> 
> (Updated July 28, 2015, 2:39 a.m.)
> 
> 
> Review request for samza, Yi Pan (Data Infrastructure), Navina Ramesh, and Naveen Somasundaram.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> After a job is submitted, it might need some configuration change, specifically it might
need more containers. In SAMZA-704 a tool is being added to write to the coordinator stream
(CoordinatorStreamWriter).  This tool can be used to write new configurations to the coordinator
stream. However, another tool (ConfigManager) is needed to read the config changes and react
to them, which is the goal of this task. This tool should be brought up after the job is submitted
and read any config changes added to the coordinator stream, and react to each accordingly.

> 
> This tool, called the Config Manager, is focusing on handling container changs by reacting
to set-config massages with key "yarn.container.count". 
> 
> The config manager is a separate standa alone module, that should be brought up separately
after the submission of a job. Therefore, you have to add two configurations to the input
config file:
> 1. yarn.rm.address= <ip of resource manager in yarn. ex: localhost >
> 2. yarn.rm.port= <the port of the resource manager http server. ex: 8088 >
> 
> The config manger will periodically poll the coordinator stream to see if there are any
new messages. This period is set to 100 ms by deafualt. However, it can be configured by adding
configManager.polling.interval=<polling interval> to the input config file. Thus, overal
the command to run the config manager along with the job would be:
> 
> 
> <path to samza deployment>/bin/run-config-manager.sh --config-factory=<config
factory> --config-path=<path to config file of a job>
> 
> 
> Diffs
> -----
> 
>   build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 
>   checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c 
>   gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 
>   samza-autoscaling/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java
PRE-CREATION 
>   samza-autoscaling/src/main/java/org/apache/samza/autoScaling/utils/YarnUtil.java PRE-CREATION

>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2

>   samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala
ea702a919348305ff95ce0b4ca1996a13aff04ec 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterService.scala ce88698c12c4bf6f4cf128f92d60b0b9496997d7

>   settings.gradle 19bff971ad221084dac10d3f7f3facfa42b829a7 
> 
> Diff: https://reviews.apache.org/r/36006/diff/
> 
> 
> Testing
> -------
> 
> Tested with hello samza and works properly.
> 
> 
> Thanks,
> 
> Shadi A. Noghabi
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message