tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@hortonworks.com>
Subject RE: Using ByteBuffers for Configuration
Date Fri, 26 Jul 2013 01:17:24 GMT
Any user related stuff, including config, is expected to be passed as a
user payload via the Vertex API. For the MR case, the only payload that
needs to get sent is Configuration and so that is serialized to
userpayload and sent over. Not quite sure how your impl of the local mode
handles these semantics.

-----Original Message-----
From: Achal Soni [mailto:asoni@twitter.com]
Sent: Thursday, July 25, 2013 5:36 PM
To: dev@tez.incubator.apache.org
Subject: Using ByteBuffers for Configuration

Hello All,

I am slightly concerned about using ByteBuffers as means for localizing
the per vertex configuration.

I thought the purpose of the byte buffers were to allow Pig/Hive to send
additional information (not necessarily in the form of Configuration
objects) to the vertices. This seems to defeat the purpose because we are
using it exclusively to deserialize the configuration object for the

I am running into this issue at the initialize method of MRRuntimeTask. In
local mode, I am setting some additional configs in the Configuration
object for that vertex and passing that forward, so it should propagate in
the following method:

initialize(Configuration conf, byte[] userPayload,

      Master master)

But now I see that I have to instead deserialize the user payload myself,
set the configs there, and then reserialize them. This is not a big change
or anything but it got my thinking.

Thoughts on this would be great!

- Achal

View raw message