tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <rbalamo...@apache.org>
Subject Re: Replacing tez task scheduler
Date Wed, 21 Dec 2016 02:38:33 GMT
E.g task scheduler in hive is llap.



(This is for Harish's point)


On Wed, Dec 21, 2016 at 7:46 AM, Harish JP <hjp@hortonworks.com> wrote:

> Hi Dharmesh,
> TaskScheduler is not picked up via config. It has to be setup in the
> TezClient using setServicePluginDescriptor. And dag/vertex execution
> context should be configured to use this task scheduler.
> What you ask is very different from what TaskScheduler in Tez does. The
> YarnTaskSchedulerService does not allocate containers, this is actually
> done by the Yarn RM. The yarn task scheduler only allocates the Tez task to
> these containers and also tries to reuse containers when possible.
> —
> Thanks,
> Harish
> On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <dhkakadia@gmail.com> wrote:
> Hi,
> I am trying to implement a new tez task scheduler which tries to allocate
> tasks uniformly across nodes. As part of this, I am trying to understand
> how does tez decide which scheduler to use. I could not find any
> configuration, My current understanding is if its running on yarntez mode,
> it will use YarnTaskSchedulerService, and if its uber mode it will use
> LocalTaskSchedulerService. I m interested in using it with hive on tez.
> Please let me know what is the configuration for using a new scheduler.
> Thanks,
> Dharmesh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message