drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject drill git commit: add cgroup doc for DoY
Date Mon, 19 Mar 2018 23:55:58 GMT
Repository: drill
Updated Branches:
  refs/heads/gh-pages 78f7f3b8a -> 99c8231b3


add cgroup doc for DoY


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/99c8231b
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/99c8231b
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/99c8231b

Branch: refs/heads/gh-pages
Commit: 99c8231b3637a598bfdb197b79cef3f21a878d8c
Parents: 78f7f3b
Author: Bridget Bevens <bbevens@maprtech.com>
Authored: Mon Mar 19 16:54:27 2018 -0700
Committer: Bridget Bevens <bbevens@maprtech.com>
Committed: Mon Mar 19 16:54:27 2018 -0700

----------------------------------------------------------------------
 .../010-drill-on-yarn-introduction.md           |  6 +-
 ...ndix-e-using-cgroups-to-control-cpu-usage.md | 94 ++++++++++++++++++++
 .../050-starting-drill-in-distributed-mode.md   | 10 +--
 3 files changed, 102 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill/blob/99c8231b/_docs/drill-on-yarn/010-drill-on-yarn-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/drill-on-yarn/010-drill-on-yarn-introduction.md b/_docs/drill-on-yarn/010-drill-on-yarn-introduction.md
index 6cf89ad..5f89b90 100644
--- a/_docs/drill-on-yarn/010-drill-on-yarn-introduction.md
+++ b/_docs/drill-on-yarn/010-drill-on-yarn-introduction.md
@@ -1,6 +1,6 @@
 ---
 title: "Drill-on-YARN Introduction"
-date: 2018-03-18 20:11:39 UTC
+date: 2018-03-19 23:54:28 UTC
 parent: "Drill-on-YARN"
 ---  
 
@@ -20,8 +20,8 @@ distributions also provide disk settings.
 
 For memory, you first configure Drill’s memory as described below, then you inform YARN
of the Drill configuration. 
 
-Drill will use all available disk I/Os. Drill will also use all available CPU. Consider enabling
Linux
-cgroups to limit Drill's CPU usage to match the YARN vcores allocation.  
+Drill will use all available disk I/Os. Drill will also use all available CPU. Consider enabling
[Linux
+cgroups](https://drill.apache.org/docs/appendix-e-using-cgroups-to-control-cpu-usage/) to
limit Drill's CPU usage to match the YARN vcores allocation.  
 
 ##Drill-on-YARN Components  
 

http://git-wip-us.apache.org/repos/asf/drill/blob/99c8231b/_docs/drill-on-yarn/094-appendix-e-using-cgroups-to-control-cpu-usage.md
----------------------------------------------------------------------
diff --git a/_docs/drill-on-yarn/094-appendix-e-using-cgroups-to-control-cpu-usage.md b/_docs/drill-on-yarn/094-appendix-e-using-cgroups-to-control-cpu-usage.md
new file mode 100644
index 0000000..3664e13
--- /dev/null
+++ b/_docs/drill-on-yarn/094-appendix-e-using-cgroups-to-control-cpu-usage.md
@@ -0,0 +1,94 @@
+---
+title: "Appendix E: Using cgroups to Control CPU Usage"
+date:  
+parent: "Drill-on-YARN"
+---   
+
+Linux cgroups (control groups) enable you to limit system resources to defined user groups
or processes. As of Drill 1.13, you can configure a cgroup for Drill (running under YARN)
to enforce CPU limits on the Drillbit service by setting the CPU limit on each Drill node
in the /etc/cgconfig.conf file.
+
+You can set the CPU limit as a soft or hard limit, or both. The hard limit takes precedence
over the soft limit. When Drill hits the hard limit, in-progress queries may not complete.
 
+
+##CPU Limits  
+
+You set the soft and hard limits with parameters in the /etc/cgconfig.conf file. The following
sections describe the parameters for soft and hard limits.  
+
+**Soft Limit Parameter**  
+You set the soft limit with the `cpu.shares` parameter. When you set a soft limit, Drill
can exceed the CPU allocated if extra CPU is available for use on the system. Drill can continue
to use CPU until there is contention with other processes over the CPU or Drill hits the hard
limit.  
+
+**Hard Limit Parameters**  
+You set the hard limit with the `cpu.cfs_period_us` and `cpu.cfs_quota_us` parameters. The
`cpu.cfs_period_us` and `cpu.cfs_quota_us` parameters set a hard limit on the amount of CPU
time that the Drill process can use.  
+
+- **`cpu.cfs_period_us`**   
+The `cpu.cfs_quota_us` parameter specifies a segment of time (in microseconds represented
by `us` for µs) for how often the access to CPU resources should be reallocated. For example,
if tasks in a cgroup can access a single CPU for 0.2 seconds out of every 1 second, set cpu.cfs_quota_us
to 200000 and cpu.cfs_period_us to 1000000. The upper limit of the `cpu.cfs_quota_us` parameter
is 1 second and the lower limit is 1000 microseconds.    
+
+
+- **`cpu.cfs_quota_us`**  
+The `cpu.cfs_quota_us` parameter specifies the total amount of runtime (in microseconds represented
by `us` for µs) for which all tasks in the Drill cgroup can run during one period (as defined
by cpu.cfs_period_us). As soon as tasks in the Drill cgroup use the time specified by the
quota, they are throttled for the remainder of the time specified by the period and not allowed
to run until the next period. For example, if tasks in the Drill cgroup can access a single
CPU for 0.2 seconds out of every 1 second, set `cpu.cfs_quota_us` to 200000 and `cpu.cfs_period_us`
to 1000000. A value of -1 indicates that the group does not have any restrictions on CPU.
 
+
+##Before You Begin
+Each Drill node must have the libcgroup package installed to configure CPU limits for a Drill
cgroup. The libcgroup package installs the cgconfig service required to configure and manage
the Drill cgroup.
+
+You can install the libcgroup package using the `yum install` command, as shown:  
+
+       yum install libcgroup  
+
+##Configuring CPU Limits
+Complete the following steps to set a hard and/or soft limit on Drill CPU usage for the Drill
process running on the node:  
+
+1-Start the cgconfig service:  
+
+        service cgconfig start
+
+2-Add a cgroup for Drill in the /etc/cgconfig.conf file:    
+
+              group drillcpu {
+                     cpu {
+                            cpu.shares = 320;
+                            cpu.cfs_quota_us = 400000;
+                            cpu.cfs_period_us = 100000;
+                            }
+                     }  
+**Note:** The cgroup name is specific to the Drill cgroup and does not correlate with any
other configuration. You can give this group any name you prefer. The name drillcpu is used
as an example.  
+  
+In the configuration example, the `cpu.shares` parameter sets the soft limit. The other two
parameters, `cpu.cfs_quota_us` and `cpu.cfs_period_us`, set the hard limit. If you prefer
to set only one type of limit, remove the parameters that do not apply.  
+
+To set a soft limit, allocate a specific number of CPU shares to the Drill cgroup in the
configuration. Calculate the CPU shares as:  
+
+       1024 (CPU allocated to Drill/Total available CPU)
+
+In the example, CPU shares was calculated as:  
+
+       1024 (10/32) = 320
+
+
+To set a hard limit, add limits to the `cpu.cfs_quota_us` and `cpu.cfs_period_us` parameters.
In the configuration example, the Drill process can fully utilize 4 CPU.  
+
+**Note:** The hard limit parameter settings persist after each cgroup service restart. Alternatively,
you can set the parameters at the session level using the following commands:  
+
+       echo 400000 > /cgroup/cpu/drillcpu/cpu.cfs_quota_us
+       echo 100000 > /cgroup/cpu/drillcpu/cpu.cfs_period_us
+
+3-(Optional) If you want the cgconfig service to automatically restart upon system reboots,
run the following command:  
+
+       chkconfig cgconfig on  
+  
+4-Run the following command to add the Drill process ID (PID) to the /cgroup/cpu/drillcpu/cgroup.procs
file, as shown:  
+
+       echo 25809 > /cgroup/cpu/drillcpu/cgroup.procs
+
+**Note:** You must perform this step each time a Drillbit restarts.  
+
+For additional information, refer to the following documentation:  
+- [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html)
 
+- [resource_management_guide/sect-cpu-example_usage](resource_management_guide/sect-cpu-example_usage)
 
+- [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html)
+- [resource_management_guide/sec-cpu_and_memory-use_case](resource_management_guide/sec-cpu_and_memory-use_case)
+
+
+
+
+
+
+
+
+

http://git-wip-us.apache.org/repos/asf/drill/blob/99c8231b/_docs/install/050-starting-drill-in-distributed-mode.md
----------------------------------------------------------------------
diff --git a/_docs/install/050-starting-drill-in-distributed-mode.md b/_docs/install/050-starting-drill-in-distributed-mode.md
index e94b5f1..e424200 100644
--- a/_docs/install/050-starting-drill-in-distributed-mode.md
+++ b/_docs/install/050-starting-drill-in-distributed-mode.md
@@ -1,6 +1,6 @@
 ---
 title: "Starting Drill in Distributed Mode"
-date: 2018-03-19 21:55:14 UTC
+date: 2018-03-19 23:54:29 UTC
 parent: "Installing Drill in Distributed Mode"
 ---
 
@@ -82,16 +82,16 @@ To exit the Drill shell, issue the following command:
 
 `!quit`
 
-## Stopping Drill 
+## Shut Down the Drill Process on a Node
 
-You can abruptly stop the Drill process on a node, or you can have the Drill process on the
node shutdown gracefully. When you stop the Drill process on a node, active queries cannot
complete if they require additional time to complete beyond the default five second wait period.
In Drill 1.12 and later, you can use the Graceful Shutdown option, which transitions a Drillbit
into a quiescent state in which the Drill process can complete in-progress queries before
shutting down.  
+You can abruptly stop the Drill process on a node, or you can have the Drill process on the
node shutdown gracefully. When you stop the Drill process on a node, active queries cannot
complete if they require more time than the default five second wait period. In Drill 1.12
and later, you can use the Graceful Shutdown option, which transitions a Drillbit into a quiescent
state in which the Drill process can complete in-progress queries before shutting down.  
 
-###Stopping Drill   
+###Stopping the Drill Process
 To stop the Drill process on the node, issue the `drillbit.sh stop` command, as shown:  
 
        bin/drillbit.sh stop   
 
-###Graceful Shutdown  
+###Gracefully Shutting Down the Drill Process
 
 Graceful shutdown is enabled by default. You can gracefully shutdown a Drillbit from the
command line or the Drill Web Console. When you initiate a graceful shutdown from the Drill
Web Console, the console posts an alert stating that a graceful shutdown was triggered. You
can see the progress of the shutdown as the Drillbit completes queries and transitions through
the quiescent state.  
 


Mime
View raw message