tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: Minutes of the meeting with Pig team
Date Sat, 25 Jan 2014 05:03:26 GMT
The important bit would be the app server support.i think given that, Tez would be pretty easy.
Ambrose ui is not actually super mr specific. It's a dag display with some extras. 

There would certainly be work to be done there. Just suggesting it as an option -- work needs
to be done somewhere, regardless. 

On Jan 24, 2014, at 1:16 PM, Rohini Palaniswamy <rohini.aditya@gmail.com> wrote:

> bq. From a priority point of view, which is more important for Pig - a
> history UI for completed applications or an AM UI to monitor a running
> application?
>    Both are important. But I guess if there is a choice to be made,
> probably will go with history UI as you can at least look at it and debug
> later looking at the logs and config. It would be difficult to always catch
> the AM in action and difficult to debug before the run finishes.
> 
> bq. Ambrose could provide both if someone hooked it up to app history
> server.
>    I am not sure how that would work, unless Tez support is added to
> Ambrose.
> 
> Regards,
> Rohini
> 
> 
> 
> On Thu, Jan 23, 2014 at 6:00 PM, Dmitriy Ryaboy <dvryaboy@gmail.com> wrote:
> 
>> Ambrose could provide both if someone hooked it up to app history server.
>> 
>> 
>> On Thu, Jan 23, 2014 at 5:11 PM, Hitesh Shah <hitesh@apache.org> wrote:
>> 
>>> At the moment, we are looking at integrating Tez with YARN's application
>>> history server mostly for data-related integration. YARN will be
>> providing
>>> webservices that provides access to this data and therefore allow anyone
>> to
>>> build a UI. However, there are no concrete plans/timelines around the UI
>>> implementation.
>>> 
>>> From a priority point of view, which is more important for Pig - a
>> history
>>> UI for completed applications or an AM UI to monitor a running
>> application?
>>> 
>>> thanks
>>> -- Hitesh
>>> 
>>> On Jan 23, 2014, at 4:53 PM, Rohini Palaniswamy wrote:
>>> 
>>>> Bikas came to the Pig Sprint Planning and we had some good discussions
>> on
>>>> the approaches we are taking.
>>>> 
>>>> Prioritized set of requirements from the Pig Team:
>>>> 
>>>> Q1:
>>>>  - Partitioned unsorted output
>>>>  - API to start Input fetch (TEZ-668)
>>>>  - Fix for any issues with 1-1 Edge found by us as we plan to use them
>>>> for one stage in order-by.
>>>>  - UI for Tez jobs
>>>> 
>>>> Q2:
>>>>  - Advanced Memory management
>>>>       - Memory manager for inputs based on map output sizes.
>>>>       - Memory manager for outputs
>>>>     As a workaround till then we will set io.sort.mb on the input and
>>>> output descriptors ourselves manually based on number of edges.
>>>> 
>>>> Q3/Q4:
>>>> - Partial aggregator to determine number of reduces
>>>> - Split edge support - same output to multiple vertices
>>>> 
>>>> 
>>>> Hitesh,
>>>>    Bikas said you would have information on the UI for Tez as you are
>>>> driving it with YARN team. Can you brief us on what we can expect and
>>>> possibly when?
>>>> 
>>>> 
>>>> Regards,
>>>> Rohini
>> 

Mime
View raw message