tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: question for why not use tez 1.0app master pool
Date Tue, 30 Dec 2014 03:48:10 GMT
Hello qiang

The difference between a session and the earlier AM pool is that the earlier version supported
running only one DAG before the AM shutdown. A session supports running DAGs in serial.

The limitation of being able to run only one DAG in one AM meant there was a launch cost of
the AM to be accounted for. The initial implementation of an AM pool was just to try out a
proof of concept to see whether an AM pool helped remove this overhead. Over time, with the
help of sessions and container reuse, the additional performance benefit that an AM pool brought
in became less and less. Furthermore, there was a lot more complexity in terms of keeping
an AM Pool service up and running, making sure that it would work properly with YARN ( and
queue allocation/utilization ). From a security point of view, an AM pool can be problematic.
Pre-launching AMs implies that they need to either be run as a super user or all jobs submitted
to the AM have to be run as the same user.

There are probably other reasons which I might have missed. Hope the above helps. Let us know
if you need more clarifications. 

— Hitesh 

On Dec 25, 2014, at 5:54 AM, SkaterXu <xxqonline@gmail.com> wrote:

> question for why not use tez 1.0app master pool?
> i have seen docs in horton works mentioned to start a app master pool in
> 2.0doc.  but why later tez use session mode app master? what is the
> difference?
> regards,
> qiang

View raw message