samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jake Maes" <jacob.m...@gmail.com>
Subject Re: Review Request 41912: SAMZA-850: Yarn Job Validation Tool
Date Wed, 06 Jan 2016 00:18:37 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41912/#review112970
-----------------------------------------------------------

Ship it!


Ship It!

- Jake Maes


On Jan. 5, 2016, 1:21 a.m., Xinyu Liu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41912/
> -----------------------------------------------------------
> 
> (Updated Jan. 5, 2016, 1:21 a.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Navina Ramesh, and Yi Pan (Data Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> We've seen a strong need for validating running samza jobs programmatically. Consider
the following scenarios:
> 1) running samza as a service (as in SAMZA-849), which requires automated job validation
when upgrading the framework for large number of jobs;
> 2) production monitoring, automated validation can run periodically and trigger alerts;
> 3) integration testing, which can use validation tool to check the job status.
> 
> So a tool is needed to provide the checkout of job health. Minimally it will check:
> - job submission
> - app running
> - container count
> 
> It should also be able to extend and validate the job metrics through metrics system,
such as JMX. The user will be able to validate against important metrics such as process_count
or message_behind_high_watermark.
> 
> 
> Diffs
> -----
> 
>   checkstyle/import-control.xml 53cb8b447240fea08d98ccfb12ed24bec6cbf67c 
>   samza-api/src/main/java/org/apache/samza/metrics/MetricsAccessor.java PRE-CREATION

>   samza-api/src/main/java/org/apache/samza/metrics/MetricsValidationFailureException.java
PRE-CREATION 
>   samza-api/src/main/java/org/apache/samza/metrics/MetricsValidator.java PRE-CREATION

>   samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 9445a30c9e605a3623873ed09eedfd5140af98f7

>   samza-core/src/main/java/org/apache/samza/metrics/JmxMetricsAccessor.java PRE-CREATION

>   samza-core/src/main/java/org/apache/samza/metrics/JmxUtil.java PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/metrics/reporter/JmxReporter.scala e9661023a04f39d059d879fea2140cb57af3b546

>   samza-core/src/test/java/org/apache/samza/metrics/TestJmxMetricsAccessor.java PRE-CREATION

>   samza-shell/src/main/bash/validate-yarn-job.sh PRE-CREATION 
>   samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java PRE-CREATION

>   samza-yarn/src/test/java/org/apache/samza/validation/MockMetricsValidator.java PRE-CREATION

>   samza-yarn/src/test/java/org/apache/samza/validation/TestYarnJobValidationTool.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41912/diff/
> 
> 
> Testing
> -------
> 
> Unit tests added to cover all the new methods/classes.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message