samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xinyu Liu" <xi...@linkedin.com>
Subject Review Request 41912: SAMZA-850: Yarn Job Validation Tool
Date Tue, 05 Jan 2016 01:21:40 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41912/
-----------------------------------------------------------

Review request for samza, Boris Shkolnik, Navina Ramesh, and Yi Pan (Data Infrastructure).


Repository: samza


Description
-------

We've seen a strong need for validating running samza jobs programmatically. Consider the
following scenarios:
1) running samza as a service (as in SAMZA-849), which requires automated job validation when
upgrading the framework for large number of jobs;
2) production monitoring, automated validation can run periodically and trigger alerts;
3) integration testing, which can use validation tool to check the job status.

So a tool is needed to provide the checkout of job health. Minimally it will check:
- job submission
- app running
- container count

It should also be able to extend and validate the job metrics through metrics system, such
as JMX. The user will be able to validate against important metrics such as process_count
or message_behind_high_watermark.


Diffs
-----

  checkstyle/import-control.xml 53cb8b447240fea08d98ccfb12ed24bec6cbf67c 
  samza-api/src/main/java/org/apache/samza/metrics/MetricsAccessor.java PRE-CREATION 
  samza-api/src/main/java/org/apache/samza/metrics/MetricsValidationFailureException.java
PRE-CREATION 
  samza-api/src/main/java/org/apache/samza/metrics/MetricsValidator.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 9445a30c9e605a3623873ed09eedfd5140af98f7

  samza-core/src/main/java/org/apache/samza/metrics/JmxMetricsAccessor.java PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/metrics/JmxUtil.java PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/metrics/reporter/JmxReporter.scala e9661023a04f39d059d879fea2140cb57af3b546

  samza-core/src/test/java/org/apache/samza/metrics/TestJmxMetricsAccessor.java PRE-CREATION

  samza-shell/src/main/bash/validate-yarn-job.sh PRE-CREATION 
  samza-yarn/src/main/java/org/apache/samza/validation/YarnJobValidationTool.java PRE-CREATION

  samza-yarn/src/test/java/org/apache/samza/validation/MockMetricsValidator.java PRE-CREATION

  samza-yarn/src/test/java/org/apache/samza/validation/TestYarnJobValidationTool.java PRE-CREATION


Diff: https://reviews.apache.org/r/41912/diff/


Testing
-------

Unit tests added to cover all the new methods/classes.


Thanks,

Xinyu Liu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message