drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aditya Kishore" <adityakish...@gmail.com>
Subject Review Request 22013: DRILL-855: Improve work assignment parallelization
Date Thu, 29 May 2014 06:15:16 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for drill, Jacques Nadeau and Steven Phillips.

Repository: drill-git


Currently if only a small number of endpoints have data affinity, all the work units get assigned
to this small subset leaving other endpoints in the cluster underutilized.

It will lead to poor query performance since the level of parallelization is reduced below
what is otherwise possible.

This JIRA intend to modify the assignment to do a slightly more equitable distribution of
work fragments among all available nodes while slightly favoring the nodes with data affinity.

This *favor* is controlled by the setting "drill.exec.work.affinity.factor" (defaults to 1.2f).

As an example, let's say that there are 2 endpoints of which only one is with affinity and
10 work fragments. In the current scheme of things, all of these work fragments will get assigned
to the single node with affinity. With the modified scheme and with default affinity factor
of 1.2, it will get 20% more assignment that it would get under equal distribution (i.e. 6:4
instead of 5:5).


  distribution/src/resources/drill-override.conf a9316a9 
  exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java d9e0833 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java f3059ae

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/SimpleExecPlanner.java 4da6500

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 8602bf0

  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java eb1d738 
  exec/java-exec/src/main/resources/drill-module.conf 26205bd 
  exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 1b38dce

Diff: https://reviews.apache.org/r/22013/diff/



Aditya Kishore

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message