tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <seth.siddha...@gmail.com>
Subject Re: TEZ-398: Changes to the underlying Task/Input/Output/Processor engine layer
Date Fri, 30 Aug 2013 17:14:47 GMT
Have created a branch for this TEZ-398.

Just to be clear, the design is likely to evolve as it sees the light of
day over the next week or so. There are also aspects which are not fully
fleshed out. At the end of this exercise, it should be possible to write a
processor which takes multiple Inputs - e.g. an input which reads MapReduce
FileFormat based input and another input which just receives non sorted
key-values generated by a previous vertex.


On Thu, Aug 29, 2013 at 4:58 PM, Hitesh Shah <hitesh@apache.org> wrote:

> Hi folks,
>
> Siddharth, Bikas and I have been working on looking at how to
> re-factor/clean up the tez-engine layer ( i.e the Input/Output/Processor
> and Task constructs ) to allow non-MR processors to be built and allow easy
> introduction of new inputs and outputs.
>
> I posted an initial design/overview draft on TEZ-398 (
> https://issues.apache.org/jira/secure/attachment/12600691/TEZ-398-Engine-Design.pdf).
Please feel free to add your comments to the jira. If you are interested
> in working on any parts of the changes, comment on either the jira and/or
> mailing lists.  If there are folks interested, we can set up a meeting
> where we can discuss the proposed design.
>
> Given that these changes are going to make the current implementation
> unstable for folks that are starting to use to tez ( in hive/pig ), we plan
> to create a new branch ( likely to be named TEZ-398 ) to work on these
> engine changes.
>
> thanks
> -- Hitesh
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message