drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject [drill] branch gh-pages updated: edit monitoring metrics page to include additional metrics
Date Tue, 13 Nov 2018 23:18:32 GMT
This is an automated email from the ASF dual-hosted git repository.

bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git


The following commit(s) were added to refs/heads/gh-pages by this push:
     new fa4525d  edit monitoring metrics page to include additional metrics
fa4525d is described below

commit fa4525d06d0d2b8302bf4778c2f2f9cd1bb22616
Author: Bridget Bevens <bbevens@maprtech.com>
AuthorDate: Tue Nov 13 15:17:48 2018 -0800

    edit monitoring metrics page to include additional metrics
---
 _docs/performance-tuning/070-monitoring-metrics.md | 65 +++++++++++++---------
 1 file changed, 40 insertions(+), 25 deletions(-)

diff --git a/_docs/performance-tuning/070-monitoring-metrics.md b/_docs/performance-tuning/070-monitoring-metrics.md
index bc52270..48004e5 100644
--- a/_docs/performance-tuning/070-monitoring-metrics.md
+++ b/_docs/performance-tuning/070-monitoring-metrics.md
@@ -1,12 +1,12 @@
 ---
 title: "Monitoring Metrics"
-date: 2018-11-10
+date: 2018-11-13
 parent: "Performance Tuning"
 --- 
 
-The Metrics page in the Drill Web UI (`http(s)://<drillbit-ip-address>:8047/metrics`)
lists JVM and operating system metrics. You can use these metrics to debug the state of the
cluster. The Drill-specific metrics are prepended with `drill`, for example `drill.fragments.running`.
The other metrics relate specifically to the JVM.
+The Metrics page in the Drill Web UI (`http(s)://<drillbit-ip-address>:8047/metrics`)
lists JVM, operating system, and certain Drill-specific metrics. You can use these metrics
to debug the state of the cluster. The Drill-specific metrics are prepended with `drill`,
for example `drill.fragments.running`. 
 
-Drill uses JMX ([Java Management Extensions](https://docs.oracle.com/javase/tutorial/jmx/))
to monitor queries at runtime. JMX provides the architecture to dynamically manage and monitor
applications. JMX collects Drill system-level metrics that you can see in the Metrics tab
in the Drill Web UI or a remote JMX monitoring tool, such as JConsole or the VisualVM + MBeans
plugin.   
+Drill uses JMX ([Java Management Extensions](https://docs.oracle.com/javase/tutorial/jmx/))
to expose metrics at runtime. JMX provides the architecture to dynamically manage and monitor
applications. JMX collects Drill system-level metrics that you can see in the Metrics tab
in the Drill Web UI or a remote JMX monitoring tool, such as JConsole or the VisualVM + MBeans
plugin.   
 
 Metrics collected by JMX are divided into the following categories on the Metrics page in
the Drill Web UI:  
 
@@ -17,7 +17,7 @@ A counter is a snapshot of the count of metrics at a particular point in
time. (
 * **Histograms**    
 A histogram measures the statistical distribution of values in a stream of data.  
 * **Meters**  
-A meter measures the rate of events over time, for example requests per second.  
+A meter measures the rate of events over time, for example requests per second. Drill currently
does not use meters to report system-level metrics.  
 * **Timers**  
 A timer measures the rate that a particular piece of code is called and the distribution
of its duration.  
 
@@ -46,50 +46,65 @@ The following table lists the Drill-specific metrics in the Gauges section
of th
 | drill.fragments.running              | The number of query   fragments currently running
in the drillbit.                                                                         
                                                                                         
                                                                                         
                                                                                         
                                          [...]
 | drill.allocator.root.used            | The amount of memory   (in bytes) used by the internal
memory allocator.                                                                        
                                                                                         
                                                                                         
                                                                                         
                                     [...]
 | drill.allocator.root.peak            | The peak amount of   memory (in bytes) used by the
internal memory allocator.                                                               
                                                                                         
                                                                                         
                                                                                         
                                         [...]
-| drill.allocator.rpc.bit.control.peak |  The maximum amount of   bytes used across all outgoing
and incoming control connections for this   Drillbit at this moment.                     
                                                                                         
                                                                                         
                                                                                         
                                    [...]
-| drill.allocator.rpc.bit.control.used | The total number of bytes currently used across
all outgoing   and incoming control connections for this Drillbit.                       
                                                                                         
                                                                                         
                                                                                         
                                            [...]
-| drill.allocator.rpc.bit.data.peak    | The maximum amount of memory used between all outgoing
and   incoming data connections for this Drillbit up to this moment.                     
                                                                                         
                                                                                         
                                                                                         
                                     [...]
-| drill.allocator.rpc.bit.data.used    |  The total amount of   memory used between all outgoing
and incoming data connections tor this   Drillbit.                                       
                                                                                         
                                                                                         
                                                                                         
                                    [...]
-| drill.allocator.rpc.bit.user.peak    | The maximum amount of memory used across all incoming
Drill   client connections to this Drillbit up to this moment.                           
                                                                                         
                                                                                         
                                                                                         
                                      [...]
-| drill.allocator.rpc.bit.user.used    | The total amount of memory used across all incoming
Drill   client connections to this Drillbit.                                             
                                                                                         
                                                                                         
                                                                                         
                                        [...]
-| drill.allocator.huge.size            |       Total size in bytes of huge (greater than
  16MB) direct buffers allocated until now                                               
                                                                                         
                                                                                         
                                                                                         
                                            [...]
-| drill.allocator.huge.count           | Number of allocations done for direct buffer of
size greater   than 16MB. Each of these allocation happens from OS which comes with an   overhead
rather from Netty's buffer pool.                                                         
                                                                                         
                                                                                         
                                    [...]
-| drill.allocator.normal.count         | Number of allocations done for direct buffer of
size less than   equal to 16MB. Each of these allocation happens from Netty's buffer pool.
  This counter is only updated in debug environment when asserts are enabled to   avoid overhead
for each allocation during normal execution.                                             
                                                                                         
                                    [...]
-| drill.allocator.normal.size          | Total size in bytes of normal (less than and equal
to 16MB)   direct buffers allocated until now. This counter is only updated in debug   environment
when asserts are enabled to avoid overhead for each allocation   during normal execution.
                                                                                         
                                                                                         
                                [...]
-| drill.allocator.normal.size          | Total size in bytes of normal (less than and equal
to 16MB)   direct buffers allocated until now. This counter is only updated in debug   environment
when asserts are enabled to avoid overhead for each allocation   during normal execution.
                                                                                         
                                                                                         
                                [...]
-| count                                | The number of   live threads, including daemon and
non-daemon threads.                                                                      
                                                                                         
                                                                                         
                                                                                         
                                         [...]
+| drill.allocator.rpc.bit.control.peak | The maximum amount of   bytes used across all outgoing
and incoming control connections for this   Drillbit at this moment.                     
                                                                                         
                                                                                         
                                                                                         
                                     [...]
+| drill.allocator.rpc.bit.control.used | The total number of   bytes currently used across
all outgoing and incoming control connections for   this Drillbit.                       
                                                                                         
                                                                                         
                                                                                         
                                          [...]
+| drill.allocator.rpc.bit.data.peak    | The maximum amount of   memory used between all
outgoing and incoming data connections for this   Drillbit up to this moment.            
                                                                                         
                                                                                         
                                                                                         
                                            [...]
+| drill.allocator.rpc.bit.data.used    | The total amount of   memory used between all outgoing
and incoming data connections tor this   Drillbit.                                       
                                                                                         
                                                                                         
                                                                                         
                                     [...]
+| drill.allocator.rpc.bit.user.peak    | The maximum amount of   memory used across all incoming
Drill client connections to this Drillbit up   to this moment.                           
                                                                                         
                                                                                         
                                                                                         
                                    [...]
+| drill.allocator.rpc.bit.user.used    | The total amount of   memory used across all incoming
Drill client connections to this Drillbit.                                               
                                                                                         
                                                                                         
                                                                                         
                                      [...]
+| drill.allocator.huge.size            | Total size in bytes of   huge (greater than 16MB)
direct buffers allocated until now                                                       
                                                                                         
                                                                                         
                                                                                         
                                          [...]
+| drill.allocator.huge.count           | Number of allocations   done for direct buffer of
size greater than 16MB. Each of these allocation   happens from OS which comes with an overhead
rather from Netty's buffer pool.                                                         
                                                                                         
                                                                                         
                                    [...]
+| drill.allocator.normal.count         | Number of allocations   done for direct buffer of
size less than equal to 16MB. Each of these   allocation happens from Netty's buffer pool.
This counter is only updated in   debug environment when asserts are enabled to avoid overhead
for each   allocation during normal execution.                                           
                                                                                         
                                    [...]
+| drill.allocator.normal.size          | Total size in bytes of   normal (less than and equal
to 16MB) direct buffers allocated until now. This   counter is only updated in debug environment
when asserts are enabled to   avoid overhead for each allocation during normal execution.
                                                                                         
                                                                                         
                                [...]
+| count                                | The number of live   threads, including daemon and
non-daemon threads.                                                                      
                                                                                         
                                                                                         
                                                                                         
                                         [...]
 | heap.used                            | The amount of heap   memory (in bytes) used by the
JVM.                                                                                     
                                                                                         
                                                                                         
                                                                                         
                                         [...]
 | non-heap.used                        | The amount of non-heap   memory (in bytes) used
by the JVM.                                                                              
                                                                                         
                                                                                         
                                                                                         
                                            [...]
 | fd.usage                             | The ratio of used file   descriptors to total file
descriptors on *nix systems.                                                             
                                                                                         
                                                                                         
                                                                                         
                                         [...]
 | direct.used                          | The amount of direct   memory (in bytes) used by
the JVM. This metric is useful for debugging Drill   issues.                             
                                                                                         
                                                                                         
                                                                                         
                                           [...]
 | runnable.count                       | The number of threads   executing an action in the
JVM. This metric is useful for debugging Drill   issues.                                 
                                                                                         
                                                                                         
                                                                                         
                                         [...]
 | waiting.count                        | The number of threads   waiting to execute. Typically,
threads waiting on other threads to perform an   action. This metric is useful for debugging
Drill issues.                                                                            
                                                                                         
                                                                                         
                                  [...]
-| load.avg                             | Returns the "recent cpu usage" for the Drillbit
  process. This value is a double in the [0.0,1.0] interval. A value of 0.0   means that none
of the CPUs were running threads from the Drillbit process   during the recent period of time
observed, while a value of 1.0 means that   all CPUs were actively running threads from the
Drillbit process 100% of the   time during the recent period being observed. Threads from
the Drillbit   process includes t [...]
-|       uptime                         | Total uptime of Drillbit JVM in miliseconds. See
[getUptime()](https://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime()).
                                                                                         
                                                                                         
                                                                                         
                       [...]
+| load.avg                             | Returns the   "recent cpu usage" for the Drillbit
process. This value is a double   in the [0.0,1.0] interval. A value of 0.0 means that none
of the CPUs were   running threads from the Drillbit process during the recent period of time
  observed, while a value of 1.0 means that all CPUs were actively running   threads from
the Drillbit process 100% of the time during the recent period   being observed. Threads from
the Drillbit process includes t [...]
+| uptime                               | Total uptime of   Drillbit JVM in miliseconds. See
[getUptime()](https://docs.oracle.com/javase/7/docs/api/java/lang/management/RuntimeMXBean.html#getUptime()).
                                                                                         
                                                                                         
                                                                                         
                     [...]
 
 ## Counters  
 
 The following table lists the Drill-specific metrics in the Counters section of the Metrics
page:   
 
-| Metric                  | Description                                                 
                                 |
-|-------------------------|-----------------------------------------------------------------------------------------------|
-| drill.queries.running   | The number of queries running for which this drillbit is the
Foreman.                         |
-| drill.queries.completed | The number of queries completed, canceled, or failed for which
this drillbit was the Foreman. |   
+| Metric                                    | Description                               
                                                                                         
                                     |
+|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| drill.connections.rpc.control.encrypted   | The   total number of encrypted incoming and
outgoing control connections to and   from this Drillbit. This includes both the control client
and server   connections.    |
+| drill.connections.rpc.control.unencrypted | The   total number of unencrypted incoming
and outgoing control connections to and   from this Drillbit. This includes both the control
client and server   connections.  |
+| drill.connections.rpc.data.encrypted      | The   total number of encrypted incoming and
outgoing data connections to and from   this Drillbit. This includes both the data client
and data server   connections.     |
+| drill.connections.rpc.data.unencrypted    | The   total number of unencrypted incoming
and outgoing data connections to and   from this Drillbit. This includes both the data client
and data server   connections.   |
+| drill.connections.rpc.user.encrypted      | The   total number of encrypted connections
from the all the Drill clients to this   Drillbit.                                       
                                    |
+| drill.connections.rpc.user.unencrypted    | The   total number of unencrypted connections
from the all the Drill clients to   this Drillbit.                                       
                                  |
+| drill.queries.canceled                    | The   number of canceled queries for which
this Drillbit was the Foreman.                                                           
                                     |
+| drill.queries.completed                   | The number of queries completed, canceled,
or   failed for which this Drillbit was the Foreman.                                     
                                     |
+| drill.queries.enqueued                    | The   number of queries waiting in one of the
configured queues for which this Drillbit is the Foreman.                                
                                                                                         
                      |
+| drill.queries.failed                      | The   number of failed queries for which this
Drillbit was the Foreman.                                                                
                                  |
+| drill.queries.planning                    | The   number of queries that are in the planning
stage for which the Drillbit is   the Foreman.                                           
                               |
+| drill.queries.running                     | The number of queries running for which this
  Drillbit is the Foreman.                                                               
                                   |
+| drill.queries.succeeded                   | The   number of successful queries for which
this Drillbit was the Foreman.                                                           
                                   |   
 
 
 ## Histograms
 
 The following table lists the Drill-specific metrics in the Histograms section of the Metrics
page:  
 
-| Metric                      | Description                                             
                                                                                         
                                                                                         
                                                                                         
                           |
+| Reporting Class                      | Description                                    
                                                                                         
                                                                                         
                                                                                         
                                    |
 |-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | drill.allocator.huge.hist   | Displays the distribution of allocation of huge buffers up
to the current   time. Like count, it specifies number of huge buffer allocations completed
so   far. Max/Min specifies maximum/minimum size in bytes of the huge buffer   allocated.
Mean and other percentiles show the distribution of the huge   buffer allocation size in bytes.
                 |
 | drill.allocator.normal.hist | Displays the distribution of allocation of the normal size
buffers up to   the current time. Like count, it specifies the number of normal buffer   allocations
completed so far. Max/Min specifies maximum/minimum size in bytes   of the normal buffer allocated.
Mean and other percentiles show the   distribution of normal buffer allocation size in bytes.
|
 
 ## Meters  
-No Drill-specific meters.
+
+Not available. 
 
 
 
 ## Timers  
-No Drill-specific timers.
\ No newline at end of file
+  
+| Reporting Class                                                         | Description 
                                                                                         
                                                                                         
                          |
+|---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| org.apache.drill.exec.cache.VectorAccessibleSerializable.writerTime       | Measures the
distribution of the   time taken to serialize a record batch to the output stream. Mainly
used to   measure the time taken to spill a record batch.                                
                          |
+| org.apache.drill.exec.store.schedule.BlockMapBuilder.blockMapBuilderTimer | Measures the
distribution of the   time taken to build a mapping of block locations for a given file byte
range.   Mainly used during the planning phase to determine a set of endpoints where   all
the data is located.  |
\ No newline at end of file


Mime
View raw message