tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: Question on Event
Date Fri, 27 Jun 2014 17:33:27 GMT
Hi Jeff, 

Events are a control-flow mechanism used by Tez. They are a way for user-land code to communicate
with either the framework or other parts of user-land code. In addition to this, events are
also used by the framework to propagate information to other parts of the framework. All events
that user-land code can generate are available as part of the tez.runtime.api ( such as DataMovement
or InputFailedError events ). 

A DataMovement event is used by an Output to send information about the data that it has generated
to a downstream Input. The event structure itself contains only a byte array as what kind
of information is being passed between a pair of Output/Input is implementation specific.
All that the framework does is route the event to the correct downstream task based on the
type of the edge that is between the 2 vertices.

There are different types of events based on what functionality/use they provide. The payloads
of these events is usually a byte array/buffer which is user-code specific and not needed
to be interpreted by the framework. Given this, in most scenarios, a custom event should not
be needed. If you have any particular use-case for which existing events are not sufficient,
lets discuss. 

Thanks for bringing this topic up. Our documentation on events is quite minimal ( non-existent

— Hitesh

On Jun 27, 2014, at 12:28 AM, Jianfeng (Jeff) Zhang <jzhang@hortonworks.com> wrote:

> Hi all,
> I notice that class Event is used by both Input, Output, Processor and
> VertexManagerPlugin.
> But IMHO it's not clear for users on how to use them and user don't have
> full control on the events. Most of the logic is in tez-dag ( such as
> whether sending it to TezChild,  it need, How to serialize/ deserialize
> these events ) It's hard to add customized events, although it may be a
> rare requirement .
> I am thinking that in the Event class, we should provide some methods that
> allow user to specify these attributes ( may need other code change ). My
> purpose is to help user understand how to use these event. What's your
> thinking on this ? Thanks
> Best Regards,
> Jeff Zhang
> -- 
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

View raw message