drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject drill-site git commit: drill 1.13 doc updates and edits
Date Mon, 26 Mar 2018 18:05:34 GMT
Repository: drill-site
Updated Branches:
  refs/heads/asf-site b38c239af -> e38c2bba1


drill 1.13 doc updates and edits


Project: http://git-wip-us.apache.org/repos/asf/drill-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill-site/commit/e38c2bba
Tree: http://git-wip-us.apache.org/repos/asf/drill-site/tree/e38c2bba
Diff: http://git-wip-us.apache.org/repos/asf/drill-site/diff/e38c2bba

Branch: refs/heads/asf-site
Commit: e38c2bba14d34a6d2425479211e8552082fdae7d
Parents: b38c239
Author: Bridget Bevens <bbevens@maprtech.com>
Authored: Mon Mar 26 11:05:14 2018 -0700
Committer: Bridget Bevens <bbevens@maprtech.com>
Committed: Mon Mar 26 11:05:14 2018 -0700

----------------------------------------------------------------------
 docs/configuring-drill-memory/index.html       |  6 +--
 docs/data-type-conversion/index.html           | 21 ++++----
 docs/parquet-filter-pushdown/index.html        | 53 ++++++++++++---------
 docs/partition-pruning-introduction/index.html | 11 ++++-
 docs/supported-data-types/index.html           |  4 +-
 feed.xml                                       |  4 +-
 6 files changed, 60 insertions(+), 39 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/configuring-drill-memory/index.html
----------------------------------------------------------------------
diff --git a/docs/configuring-drill-memory/index.html b/docs/configuring-drill-memory/index.html
index 476d195..10d18d9 100644
--- a/docs/configuring-drill-memory/index.html
+++ b/docs/configuring-drill-memory/index.html
@@ -1216,7 +1216,7 @@
 
     </div>
 
-     Mar 22, 2018
+     Mar 26, 2018
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
@@ -1279,7 +1279,7 @@ export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-&quot;&lt;limit&gt;&qu
 
 <p><strong>Example</strong>  </p>
 
-<p>If a system has 48GB of free memory and you set the following parameters in drill-env.sh:
 </p>
+<p>If a system has 48GB of memory and you set the following parameters in drill-env.sh:
 </p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">
  DRILLBIT_MAX_PROC_MEM=25%
    DRILL_HEAP=8G
    DRILL_MAX_DIRECT_MEMORY=10G
@@ -1290,7 +1290,7 @@ export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-&quot;&lt;limit&gt;&qu
    [ERROR] Unable to start Drillbit due to memory constraint violations Total Memory Requested
: 19 GB 
    Check and modify the settings or increase the maximum amount of memory permitted.
 </code></pre></div>
-<p>If DRILLBIT_MAX_PROC_MEM is increased to 50%; the Drillbit starts up with the following
warnings:  </p>
+<p>If <code>DRILLBIT_MAX_PROC_MEM</code> is increased to 50%; the Drillbit
starts up with the following warnings:  </p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">
  [WARN] 50% of the system memory (48 GB) translates to 24 GB.
    [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB.

    You can increase your existing memory configuration to use this extra memory.  

http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/data-type-conversion/index.html
----------------------------------------------------------------------
diff --git a/docs/data-type-conversion/index.html b/docs/data-type-conversion/index.html
index ea61807..7685349 100644
--- a/docs/data-type-conversion/index.html
+++ b/docs/data-type-conversion/index.html
@@ -1218,7 +1218,7 @@
 
     </div>
 
-     
+     Mar 26, 2018
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
@@ -1377,13 +1377,18 @@ CONVERT_FROM(column, type)
 
 <h3 id="convert_to-and-convert_from-usage-notes">CONVERT_TO and CONVERT_FROM Usage
Notes</h3>
 
-<p>CONVERT_FROM and CONVERT_TO methods transform a known binary representation/encoding
to a Drill internal format. Use CONVERT_TO and CONVERT_FROM instead of the CAST function for
converting binary data types. CONVERT_TO/FROM functions work for data in a binary representation
and are more efficient to use than CAST. </p>
-
-<p>Drill can optimize scans of HBase tables when you use the *_BE encoded types shown
in section  <a href="/docs/supported-data-types/#data-types-for-convert_to-and-convert_from-functions">&quot;Data
Types for CONVERT_TO and CONVERT_FROM Functions&quot;</a> on big endian-encoded
data. You need to use the HBase storage plugin and query data as described in <a href="/docs/querying-hbase">&quot;Querying
Hbase&quot;</a>. To write Parquet binary data, convert SQL data <em>to</em>
binary data and store the data in a Parquet table while creating a table as a selection (CTAS).</p>
-
-<p>CONVERT_TO also converts an SQL data type to complex types, including HBase byte
arrays, JSON and Parquet arrays, and maps. CONVERT_FROM converts from complex types, including
HBase arrays, JSON and Parquet arrays and maps to an SQL data type. </p>
-
-<p>You can use <a href="/docs/data-type-conversion/#string_binary-function">STRING_BINARY</a>
and <a href="/docs/data-type-conversion/#binary_string-function">BINARY_STRING</a>
custom Drill functions with CONVERT_TO and CONVERT_FROM to get meaningful results.</p>
+<ul>
+<li><p>CONVERT_FROM and CONVERT_TO methods transform a known binary representation/encoding
to a Drill internal format. Use CONVERT_TO and CONVERT_FROM instead of the CAST function for
converting binary data types. CONVERT_TO/FROM functions work for data in a binary representation
and are more efficient to use than CAST. </p></li>
+<li><p>Drill can optimize scans on HBase tables when you use the *_BE encoded
types shown in section  <a href="/docs/supported-data-types/#data-types-for-convert_to-and-convert_from-functions">&quot;Data
Types for CONVERT_TO and CONVERT_FROM Functions&quot;</a> on big endian-encoded
data. You need to use the HBase storage plugin and query data as described in <a href="/docs/querying-hbase">&quot;Querying
Hbase&quot;</a>. To write Parquet binary data, convert SQL data <em>to</em>
binary data and store the data in a Parquet table while creating a table as a selection (CTAS).
 </p></li>
+<li><p>CONVERT_TO also converts an SQL data type to complex types, including
HBase byte arrays, JSON and Parquet arrays, and maps. CONVERT_FROM converts from complex types,
including HBase arrays, JSON and Parquet arrays and maps to an SQL data type. </p></li>
+<li><p>You can use <a href="/docs/data-type-conversion/#string_binary-function">STRING_BINARY</a>
and <a href="/docs/data-type-conversion/#binary_string-function">BINARY_STRING</a>
custom Drill functions with CONVERT_TO and CONVERT_FROM to get meaningful results.  </p></li>
+<li><p>Drill 1.13 and later supports <a href="/docs/json-data-model/">NaN
and Infinity values as numeric data types</a>. 
+You can use the convert_to and convert_from functions in queries on JSON data with NaN and
Infinity values, as shown in the following query examples:  </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">
   select convert_fromJSON(&#39;{&quot;num&quot;: 55, &quot;nan&quot;:
NaN, &quot;inf&quot;: -Infinity}&#39;); 
+    select convert_fromJSON(jsonColumn) from mysql.someTable;  
+    select string_binary(convert_toJSON(convert_fromJSON(jsonColumn) from mysql.someTable;
+</code></pre></div></li>
+</ul>
 
 <h3 id="conversion-of-data-types-examples">Conversion of Data Types Examples</h3>
 

http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/parquet-filter-pushdown/index.html
----------------------------------------------------------------------
diff --git a/docs/parquet-filter-pushdown/index.html b/docs/parquet-filter-pushdown/index.html
index fdde36b..5f69651 100644
--- a/docs/parquet-filter-pushdown/index.html
+++ b/docs/parquet-filter-pushdown/index.html
@@ -1216,7 +1216,7 @@
 
     </div>
 
-     Nov 21, 2016
+     Mar 26, 2018
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
@@ -1226,6 +1226,13 @@
 
 <p>Parquet filter pushdown is enabled by default. When a query contains a filter expression,
you can run the <a href="/docs/explain-commands/">EXPLAIN PLAN command</a> to
see if Drill applies Parquet filter pushdown to the query. You can enable and disable this
feature using the <a href="/docs/alter-system/">ALTER SYSTEM|SESSION SET</a> command
with the <code>planner.store.parquet.rowgroup.filter.pushdown</code> option. 
</p>
 
+<p>As of Drill 1.13, the query planner in Drill can apply project push down, filter
push down, and partition pruning to star queries in common table expressions (CTEs), views,
and subqueries, for example:  </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">
  select col1 from (select * from t)  
+</code></pre></div>
+<p>When a CTE, view, or subquery contains a star filter condition, the query planner
in Drill can apply the filter and prune extraneous data, further reducing the amount of data
that the scanner reads and improving performance. </p>
+
+<p><strong>Note:</strong> Currently, Drill only supports pushdown for simple
star subselect queries without filters. See <a href="https://www.google.com/url?q=https://issues.apache.org/jira/browse/DRILL-6219&amp;sa=D&amp;ust=1522084453671000&amp;usg=AFQjCNFXp-nWMRXzM466BSRFlV3F63_ZYA">DRILL-6219</a>
for more information.  </p>
+
 <h2 id="how-parquet-filter-pushdown-works">How Parquet Filter Pushdown Works</h2>
 
 <p>Drill applies Parquet filter pushdown during the query planning phase. The query
planner in Drill performs Parquet filter pushdown by evaluating the filter expressions in
the query. If no filter expression exists, the underlying scan operator reads all of the data
in a Parquet file and then sends the data to operators downstream. When filter expressions
exist, the planner applies each filter and prunes the data, reducing the amount of data that
the scanner and Parquet reader must read.</p>
@@ -1273,44 +1280,46 @@
 
 <h2 id="support">Support</h2>
 
-<p>The following table lists the supported and unsupported clauses, operators, data
types, and scenarios for Parquet filter pushdown:  </p>
+<p>The following table lists the supported and unsupported clauses, operators, data
types, function, and scenarios for Parquet filter pushdown:  </p>
+
+<p><strong>Note:</strong> An asterisk (*) indicates support as of Drill
1.13</p>
 
 <table><thead>
 <tr>
 <th></th>
-<th>Supported</th>
-<th>Not Supported</th>
+<th><strong>Supported</strong></th>
+<th><strong>Not Supported</strong></th>
 </tr>
 </thead><tbody>
 <tr>
-<td>Clauses</td>
-<td>WHERE,   HAVING (HAVING is supported if Drill can pass the filter through GROUP
  BY.)</td>
-<td></td>
+<td><strong>Clauses</strong></td>
+<td>WHERE, *WITH, HAVING (HAVING is   supported if Drill can pass the filter through
GROUP BY.)</td>
+<td>--</td>
 </tr>
 <tr>
-<td>Operators</td>
-<td>AND,   OR, IN (An IN list is converted to OR if the number in the IN list is within
  a certain threshold, for example 20. If greater than the threshold, pruning   cannot occur.)</td>
-<td>NOT,   ITEM (Drill does not push the filter past the ITEM operator, which is used
  for complex fields.)</td>
+<td><strong>Operators</strong></td>
+<td>AND, OR, NOT, *IS [NOT] NULL, *IS   [NOT] TRUE</td>
+<td>FALSE, IN (An IN list is converted to OR if the number in the IN   list is within
a certain threshold, for example 20. If greater than the   threshold, pruning cannot occur.)</td>
 </tr>
 <tr>
-<td>Comparison Operators</td>
-<td>&lt;&gt;,   &lt;, &gt;, &lt;=, &gt;=, =</td>
-<td>IS [NOT] NULL</td>
+<td>Comparison   Operators</td>
+<td>&lt;&gt;, &lt;, &gt;, &lt;=, &gt;=, =</td>
+<td>--</td>
 </tr>
 <tr>
-<td>Data Types</td>
-<td>INT,   BIGINT, FLOAT, DOUBLE, DATE, TIMESTAMP, TIME</td>
-<td>CHAR,   VARCHAR columns, Hive TIMESTAMP</td>
+<td><strong>Data   Types</strong></td>
+<td>INT, BIGINT, FLOAT, DOUBLE, DATE,   TIMESTAMP, TIME, *BOOLEAN (true</td>
+<td>false)</td>
 </tr>
 <tr>
-<td>Function</td>
-<td>CAST   is supported among these four numeric types only: int, bigint, float, double</td>
-<td></td>
+<td><strong>Function</strong></td>
+<td>CAST is supported among the following   types only: int, bigint, float, double,
*date, *timestamp, and *time</td>
+<td>--</td>
 </tr>
 <tr>
-<td>Other</td>
-<td>--</td>
-<td>Joins,   Files with multiple row groups, Enabled Native Hive reader</td>
+<td><strong>Other</strong></td>
+<td>Files with multiple row groups</td>
+<td>Joins, Enabled Native Hive reader</td>
 </tr>
 </tbody></table>
 

http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/partition-pruning-introduction/index.html
----------------------------------------------------------------------
diff --git a/docs/partition-pruning-introduction/index.html b/docs/partition-pruning-introduction/index.html
index c881e68..4b8c110 100644
--- a/docs/partition-pruning-introduction/index.html
+++ b/docs/partition-pruning-introduction/index.html
@@ -1218,7 +1218,7 @@
 
     </div>
 
-     Aug 15, 2016
+     Mar 26, 2018
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
@@ -1228,7 +1228,14 @@
 
 <p>As of Drill 1.8, partition pruning also applies to the Parquet metadata cache. When
data is partitioned in a directory hierarchy, Drill attempts to read the metadata cache file
from a sub-partition, based on matching filter criteria instead of reading from the top level
partition, to reduce the amount of metadata read during the query planning time. If you created
a metadata cache file in a previous version of Drill, you must issue the REFRESH TABLE METADATA
command to regenerate the metadata cache file before running queries for metadata cache pruning
to occur. See <a href="/docs/optimizing-parquet-metadata-reading/">Optimizing Parquet
Metadata Reading</a> for more information.  </p>
 
-<p>The query planner in Drill performs partition pruning by evaluating the filters.
If no partition filters are present, the underlying Scan operator reads all files in all directories
and then sends the data to operators, such as Filter, downstream. When partition filters are
present, the query planner pushes the filters down to the Scan if possible. The Scan reads
only the directories that match the partition filters, thus reducing disk I/O.</p>
+<p>The query planner in Drill performs partition pruning by evaluating the filters.
If no partition filters are present, the underlying Scan operator reads all files in all directories
and then sends the data to operators, such as Filter, downstream. When partition filters are
present, the query planner pushes the filters down to the Scan if possible. The Scan reads
only the directories that match the partition filters, thus reducing disk I/O.  </p>
+
+<p>As of Drill 1.13, the query planner in Drill can apply project push down, filter
push down, and partition pruning to star queries in common table expressions (CTEs), views,
and subqueries, for example:  </p>
+<div class="highlight"><pre><code class="language-text" data-lang="text">
  select col1 from (select * from t)  
+</code></pre></div>
+<p>When a CTE, view, or subquery contains a star filter condition, the query planner
in Drill can apply the filter and prune extraneous data, further reducing the amount of data
that the scanner reads and improving performance. </p>
+
+<p><strong>Note:</strong> Currently, Drill only supports pushdown for simple
star subselect queries without filters. See <a href="https://www.google.com/url?q=https://issues.apache.org/jira/browse/DRILL-6219&amp;sa=D&amp;ust=1522084453671000&amp;usg=AFQjCNFXp-nWMRXzM466BSRFlV3F63_ZYA">DRILL-6219</a>
for more information.</p>
 
 <h2 id="using-partitioned-drill-data">Using Partitioned Drill Data</h2>
 

http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/supported-data-types/index.html
----------------------------------------------------------------------
diff --git a/docs/supported-data-types/index.html b/docs/supported-data-types/index.html
index af064cf..f31a164 100644
--- a/docs/supported-data-types/index.html
+++ b/docs/supported-data-types/index.html
@@ -1218,7 +1218,7 @@
 
     </div>
 
-     Apr 5, 2017
+     Mar 26, 2018
 
     <link href="/css/docpage.css" rel="stylesheet" type="text/css">
 
@@ -1760,7 +1760,7 @@ Converts a string to TIMESTAMP.</li>
 <p>The <a href="/docs/data-type-conversion/#convert_to-and-convert_from">CONVERT_TO
function</a> converts data to bytes from the input type. The <a href="/docs/data-type-conversion/#convert_to-and-convert_from">CONVERT_FROM
function</a> converts data from bytes to the input type. For example, the following
CONVERT_TO function converts an integer to bytes using big endian encoding:</p>
 <div class="highlight"><pre><code class="language-text" data-lang="text">CONVERT_TO(mycolumn,
&#39;INT_BE&#39;)
 </code></pre></div>
-<p>The following table lists the data types for use with the CONVERT_TO
+<p>The following table lists the data types you can use with the CONVERT_TO
 and CONVERT_FROM functions:</p>
 
 <table><thead>

http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index 6e56d6d..83cfc33 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
 </description>
     <link>/</link>
     <atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
-    <pubDate>Thu, 22 Mar 2018 11:56:49 -0700</pubDate>
-    <lastBuildDate>Thu, 22 Mar 2018 11:56:49 -0700</lastBuildDate>
+    <pubDate>Mon, 26 Mar 2018 11:02:52 -0700</pubDate>
+    <lastBuildDate>Mon, 26 Mar 2018 11:02:52 -0700</lastBuildDate>
     <generator>Jekyll v2.5.2</generator>
     
       <item>


Mime
View raw message