tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dharmesh Kakadia <dhkaka...@gmail.com>
Subject Re: Replacing tez task scheduler
Date Wed, 21 Dec 2016 06:51:36 GMT
Thanks Harish, that makes sense. I will give it a try.

Thanks,
Dharmesh

On Tue, Dec 20, 2016 at 10:23 PM, Harish JP <hjp@hortonworks.com> wrote:

> Dharmesh,
>
> The code Rajesh pointed out was to show how to use a custom scheduler. But
> that is not a configuration option but something which has to be part of
> code which creates the Tez client. You’ll not be able to customize the
> scheduler without a code change in hive or tez. I would recommend you to
> try to modify hive source and plugin your scheduler or modify the tez
> source and use your class: https://github.com/apache/tez/
> blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/
> rm/TaskSchedulerManager.java#L550. But test it out and see if what you
> asked for works and we can figure out integration: modify existing
> scheduler, add configurable alternative to yarn scheduler, add configurable
> default scheduler, etc ...
>
> —
> Thanks,
> Harish
>
> On 21-Dec-2016, at 9:33 AM, Dharmesh Kakadia <dhkakadia@gmail.com<mailto:
> dhkakadia@gmail.com>> wrote:
>
> Thanks Harish.
> I understand YARN allocates container, I am interested in what tasks to run
> on those containers once it is available to tez task scheduler.
>
> @Rajesh I am interested in task scheduling on non-LLAP hive cluster. Is
> this currently not possible ?
>
> Thanks,
> Dharmesh
>
> On Tue, Dec 20, 2016 at 6:38 PM, Rajesh Balamohan <rbalamohan@apache.org
> <mailto:rbalamohan@apache.org>>
> wrote:
>
> E.g task scheduler in hive is llap.
>
> https://github.com/apache/hive/blob/master/llap-tez/src/
> java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>
> https://github.com/apache/hive/blob/master/ql/src/java/
> org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L289
>
> https://github.com/apache/hive/blob/master/ql/src/java/
> org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L309
> (This is for Harish's point)
>
> ~Rajesh.B
>
> On Wed, Dec 21, 2016 at 7:46 AM, Harish JP <hjp@hortonworks.com> wrote:
>
> Hi Dharmesh,
>
> TaskScheduler is not picked up via config. It has to be setup in the
> TezClient using setServicePluginDescriptor. And dag/vertex execution
> context should be configured to use this task scheduler.
>
> What you ask is very different from what TaskScheduler in Tez does. The
> YarnTaskSchedulerService does not allocate containers, this is actually
> done by the Yarn RM. The yarn task scheduler only allocates the Tez task
> to
> these containers and also tries to reuse containers when possible.
>
> —
> Thanks,
> Harish
>
> On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
> wrote:
>
> Hi,
>
> I am trying to implement a new tez task scheduler which tries to allocate
> tasks uniformly across nodes. As part of this, I am trying to understand
> how does tez decide which scheduler to use. I could not find any
> configuration, My current understanding is if its running on yarntez
> mode,
> it will use YarnTaskSchedulerService, and if its uber mode it will use
> LocalTaskSchedulerService. I m interested in using it with hive on tez.
>
> Please let me know what is the configuration for using a new scheduler.
>
> Thanks,
> Dharmesh
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message