tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Sharma <hit...@microsoft.com.INVALID>
Subject Custom routing in EdgeManager
Date Tue, 19 Sep 2017 23:24:21 GMT
Hello,


I'm looking to add a custom edge manager which allows me to route events between two vertices
using some custom protocol. For instance I want to say that DataMovementEvent(s) from the
tasks in the source vertex should be routed to the tasks in the destination vertex based on
the fact whether the tasks are in the same rack or not (or for that matter use some other
key to route events between the tasks in the two stages). To do this I implemented my own
EdgeManagerPluginOnDemand derivative but I see it has two APIs for routing the events:


routeDataMovementEventToDestination<https://tez.apache.org/releases/0.8.2/tez-api-javadocs/org/apache/tez/dag/api/EdgeManagerPluginOnDemand.html#routeDataMovementEventToDestination(int,%20int,%20int)>(int
sourceTaskIndex, int sourceOutputIndex, int destinationTaskIndex)


routeDataMovementEventToDestination<https://tez.apache.org/releases/0.8.2/tez-api-javadocs/org/apache/tez/dag/api/EdgeManagerPluginOnDemand.html#routeDataMovementEventToDestination(org.apache.tez.runtime.api.events.DataMovementEvent,%20int,%20int,%20java.util.Map)>(DataMovementEvent<https://tez.apache.org/releases/0.8.2/tez-api-javadocs/org/apache/tez/runtime/api/events/DataMovementEvent.html>
event, int sourceTaskIndex, int sourceOutputIndex, Map<http://docs.oracle.com/javase/7/docs/api/java/util/Map.html?is-external=true><Integer<http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true>,List<http://docs.oracle.com/javase/7/docs/api/java/util/List.html?is-external=true><Integer<http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html?is-external=true>>>
destinationTaskAndInputIndices)

My questions is:


  *   What's the difference between the two APIs and which one is to be used? The API with
DataMovementEvent doesn't seem to be getting called with ScatterGather edge manager and others.
  *   If this API is deprecated then it is not sufficient in my case to do the routing as
I need some more metadata, which I could have got from the DataMovementEvent payload for e.g.,
so what options do I have here?


Thanks,

Hitesh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message