tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish JP <...@hortonworks.com>
Subject Re: Replacing tez task scheduler
Date Wed, 21 Dec 2016 02:16:15 GMT
Hi Dharmesh,

TaskScheduler is not picked up via config. It has to be setup in the TezClient using setServicePluginDescriptor.
And dag/vertex execution context should be configured to use this task scheduler.

What you ask is very different from what TaskScheduler in Tez does. The YarnTaskSchedulerService
does not allocate containers, this is actually done by the Yarn RM. The yarn task scheduler
only allocates the Tez task to these containers and also tries to reuse containers when possible.


On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <dhkakadia@gmail.com> wrote:


I am trying to implement a new tez task scheduler which tries to allocate
tasks uniformly across nodes. As part of this, I am trying to understand
how does tez decide which scheduler to use. I could not find any
configuration, My current understanding is if its running on yarntez mode,
it will use YarnTaskSchedulerService, and if its uber mode it will use
LocalTaskSchedulerService. I m interested in using it with hive on tez.

Please let me know what is the configuration for using a new scheduler.


View raw message