tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish JP <...@hortonworks.com>
Subject Re: Replacing tez task scheduler
Date Wed, 21 Dec 2016 06:23:56 GMT
Dharmesh,

The code Rajesh pointed out was to show how to use a custom scheduler. But that is not a configuration
option but something which has to be part of code which creates the Tez client. You’ll not
be able to customize the scheduler without a code change in hive or tez. I would recommend
you to try to modify hive source and plugin your scheduler or modify the tez source and use
your class: https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java#L550.
But test it out and see if what you asked for works and we can figure out integration: modify
existing scheduler, add configurable alternative to yarn scheduler, add configurable default
scheduler, etc ...

—
Thanks,
Harish

On 21-Dec-2016, at 9:33 AM, Dharmesh Kakadia <dhkakadia@gmail.com<mailto:dhkakadia@gmail.com>>
wrote:

Thanks Harish.
I understand YARN allocates container, I am interested in what tasks to run
on those containers once it is available to tez task scheduler.

@Rajesh I am interested in task scheduling on non-LLAP hive cluster. Is
this currently not possible ?

Thanks,
Dharmesh

On Tue, Dec 20, 2016 at 6:38 PM, Rajesh Balamohan <rbalamohan@apache.org<mailto:rbalamohan@apache.org>>
wrote:

E.g task scheduler in hive is llap.

https://github.com/apache/hive/blob/master/llap-tez/src/
java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java

https://github.com/apache/hive/blob/master/ql/src/java/
org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L289

https://github.com/apache/hive/blob/master/ql/src/java/
org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L309
(This is for Harish's point)

~Rajesh.B

On Wed, Dec 21, 2016 at 7:46 AM, Harish JP <hjp@hortonworks.com> wrote:

Hi Dharmesh,

TaskScheduler is not picked up via config. It has to be setup in the
TezClient using setServicePluginDescriptor. And dag/vertex execution
context should be configured to use this task scheduler.

What you ask is very different from what TaskScheduler in Tez does. The
YarnTaskSchedulerService does not allocate containers, this is actually
done by the Yarn RM. The yarn task scheduler only allocates the Tez task
to
these containers and also tries to reuse containers when possible.

—
Thanks,
Harish

On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
wrote:

Hi,

I am trying to implement a new tez task scheduler which tries to allocate
tasks uniformly across nodes. As part of this, I am trying to understand
how does tez decide which scheduler to use. I could not find any
configuration, My current understanding is if its running on yarntez
mode,
it will use YarnTaskSchedulerService, and if its uber mode it will use
LocalTaskSchedulerService. I m interested in using it with hive on tez.

Please let me know what is the configuration for using a new scheduler.

Thanks,
Dharmesh
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message