drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aditya Kishore (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-855) Improve work assignment parallelization
Date Thu, 29 May 2014 06:12:01 GMT
Aditya Kishore created DRILL-855:

             Summary: Improve work assignment parallelization
                 Key: DRILL-855
                 URL: https://issues.apache.org/jira/browse/DRILL-855
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Aditya Kishore
            Assignee: Aditya Kishore

Currently if only a small number of endpoints have data affinity, all the work units get assigned
to this small subset leaving other endpoints in the cluster underutilized.

It will lead to poor query performance since the level of parallelization is reduced below
what is otherwise possible.

This JIRA intend to modify the assignment to do a slightly more equitable distribution of
work fragments among all available nodes while slightly favoring the nodes with data affinity.

This \*favor\* is controlled by the setting {{"drill.exec.work.affinity.factor"}} (defaults
to 1.2f).

As an example, let's say that there are 2 endpoints of which only one is with affinity and
10 work fragments. In the current scheme of things, all of these work fragments will get assigned
to the single node with affinity. With the modified scheme and with default affinity factor
of 1.2, it will get 20% more assignment that it would get under equal distribution (i.e. 6:4
instead of 5:5).

This message was sent by Atlassian JIRA

View raw message