samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Logvinenko <mlogvine...@gmail.com>
Subject Review Request 51633: SAMZA-1013: Add YARN Node label support
Date Mon, 05 Sep 2016 17:16:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51633/
-----------------------------------------------------------

Review request for samza.


Bugs: SAMZA-1013
    https://issues.apache.org/jira/browse/SAMZA-1013


Repository: samza


Description
-------

YARN Node labels were introduced in Hadoop version 2.6, which allows to group nodes with similar
characteristics and allows applications to specify where to run. This patch adds support for
YARN node labels in Samza.

In this implementation, node labels are defined directly in yarnConfig in YarnClusterResourceManager.
It might be better to have node labels as a part of SamzaResourceRequest and SamzaResource
classes, but org.apache.hadoop.yarn.api.records.Container class doesn't contain node label
and hence we have nothing to pass to the SamzaResource constructor in onContainersAllocated
method of YarnClusterResourceManager class.


Diffs
-----

  samza-yarn/src/main/java/org/apache/samza/config/YarnConfig.java 8f2dc48 
  samza-yarn/src/main/java/org/apache/samza/job/yarn/YarnClusterResourceManager.java 96d3d7c

  samza-yarn/src/main/scala/org/apache/samza/job/yarn/ClientHelper.scala 0998c43 

Diff: https://reviews.apache.org/r/51633/diff/


Testing
-------


Thanks,

Maxim Logvinenko


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message