tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: Minutes of the meeting with Pig team
Date Fri, 24 Jan 2014 02:00:09 GMT
Ambrose could provide both if someone hooked it up to app history server.


On Thu, Jan 23, 2014 at 5:11 PM, Hitesh Shah <hitesh@apache.org> wrote:

> At the moment, we are looking at integrating Tez with YARN's application
> history server mostly for data-related integration. YARN will be providing
> webservices that provides access to this data and therefore allow anyone to
> build a UI. However, there are no concrete plans/timelines around the UI
> implementation.
>
> From a priority point of view, which is more important for Pig - a history
> UI for completed applications or an AM UI to monitor a running application?
>
> thanks
> -- Hitesh
>
> On Jan 23, 2014, at 4:53 PM, Rohini Palaniswamy wrote:
>
> > Bikas came to the Pig Sprint Planning and we had some good discussions on
> > the approaches we are taking.
> >
> > Prioritized set of requirements from the Pig Team:
> >
> > Q1:
> >   - Partitioned unsorted output
> >   - API to start Input fetch (TEZ-668)
> >   - Fix for any issues with 1-1 Edge found by us as we plan to use them
> > for one stage in order-by.
> >   - UI for Tez jobs
> >
> > Q2:
> >   - Advanced Memory management
> >        - Memory manager for inputs based on map output sizes.
> >        - Memory manager for outputs
> >      As a workaround till then we will set io.sort.mb on the input and
> > output descriptors ourselves manually based on number of edges.
> >
> > Q3/Q4:
> >  - Partial aggregator to determine number of reduces
> >  - Split edge support - same output to multiple vertices
> >
> >
> > Hitesh,
> >     Bikas said you would have information on the UI for Tez as you are
> > driving it with YARN team. Can you brief us on what we can expect and
> > possibly when?
> >
> >
> > Regards,
> > Rohini
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message