knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lmc...@apache.org
Subject svn commit: r1596890 [1/2] - in /knox: site/books/knox-0-5-0/ trunk/books/0.5.0/
Date Thu, 22 May 2014 14:30:01 GMT
Author: lmccay
Date: Thu May 22 14:30:00 2014
New Revision: 1596890

URL: http://svn.apache.org/r1596890
Log:
updated 0.5.0 book

Modified:
    knox/site/books/knox-0-5-0/knox-0-5-0.html
    knox/site/books/knox-0-5-0/likeised
    knox/trunk/books/0.5.0/book.md
    knox/trunk/books/0.5.0/book_client-details.md
    knox/trunk/books/0.5.0/book_gateway-details.md
    knox/trunk/books/0.5.0/book_getting-started.md
    knox/trunk/books/0.5.0/book_limitations.md
    knox/trunk/books/0.5.0/book_troubleshooting.md
    knox/trunk/books/0.5.0/config.md
    knox/trunk/books/0.5.0/config_kerberos.md
    knox/trunk/books/0.5.0/config_preauth_sso_provider.md
    knox/trunk/books/0.5.0/knox_cli.md
    knox/trunk/books/0.5.0/quick_start.md
    knox/trunk/books/0.5.0/service_hbase.md
    knox/trunk/books/0.5.0/service_hive.md

Modified: knox/site/books/knox-0-5-0/knox-0-5-0.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-5-0/knox-0-5-0.html?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/site/books/knox-0-5-0/knox-0-5-0.html (original)
+++ knox/site/books/knox-0-5-0/knox-0-5-0.html Thu May 22 14:30:00 2014
@@ -21,7 +21,6 @@
   <ul>
     <li><a href="#Apache+Knox+Directory+Layout">Apache Knox Directory Layout</a></li>
     <li><a href="#Supported+Services">Supported Services</a></li>
-    <li><a href="#Configure+Sandbox+port+mapping+for+VirtualBox">Configure Sandbox port mapping for VirtualBox</a></li>
   </ul></li>
   <li><a href="#Gateway+Details">Gateway Details</a>
   <ul>
@@ -74,30 +73,25 @@
   <li>Do Hadoop with Knox</li>
 </ol><h3><a id="1+-+Requirements"></a>1 - Requirements</h3><h4><a id="Java"></a>Java</h4><p>Java 1.6 or later is required for the Knox Gateway runtime. Use the command below to check the version of Java installed on the system where Knox will be running.</p>
 <pre><code>java -version
-</code></pre><h4><a id="Hadoop"></a>Hadoop</h4><p>Knox supports Hadoop 1.x or 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment. </p><h3><a id="2+-+Download+Hadoop+2.x+VM"></a>2 - Download Hadoop 2.x VM</h3><p>The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine <a href="http://hortonworks.com/products/hdp-2/#install">Sandbox</a>. Please note Knox supports other Hadoop distributions and is configurable against a full blown Hadoop cluster. Configuring Knox for Hadoop 1.x/2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster is documented in advance deployment guide.</p><h3><a id="3+-+Download+Apache+Knox+Gateway"></a>3 - Download Apache Knox Gateway</h3><p>Download one of the distributions below from the <a href="http://www.apache.org/dyn/closer.cgi/knox">Apache mirrors</a>.</p>
+</code></pre><h4><a id="Hadoop"></a>Hadoop</h4><p>Knox 0.4.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment. </p><h3><a id="2+-+Download+Hadoop+2.x+VM"></a>2 - Download Hadoop 2.x VM</h3><p>The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine <a href="http://hortonworks.com/products/hdp-2/#install">Sandbox</a>. Please note Knox supports other Hadoop distributions and is configurable against a full blown Hadoop cluster. Configuring Knox for Hadoop 2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster is documented in advance deployment guide.</p><h3><a id="3+-+Download+Apache+Knox+Gateway"></a>3 - Download Apache Knox Gateway</h3><p>Download one of the distributions below from the <a href="http://www.apache.org/dyn/closer.cgi/knox">Apache mirrors</a>.</p>
 <ul>
-  <li>Source archive: <a href="http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip">knox-incubating-0.4.0-src.zip</a> (<a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating-src.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.md5">MD5 digest</a>)</li>
-  <li>Binary archive: <a href="http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip">knox-incubating-0.4.0.zip</a> (<a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.md5">MD5 digest</a>)</li>
-  <li>RPM package: <a href="http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm">knox-incubating-0.4.0.rpm</a> (<a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating.rpm.asc">PGP signature</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.sha">SHA1 digest</a>, <a href="http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.md5">MD5 digest</a>)</li>
+  <li>Source archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip">knox-0.4.0-src.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.md5">MD5 digest</a>)</li>
+  <li>Binary archive: <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip">knox-0.4.0.zip</a> (<a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.asc">PGP signature</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.sha">SHA1 digest</a>, <a href="http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.md5">MD5 digest</a>)</li>
 </ul><p>Apache Knox Gateway releases are available under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>. See the NOTICE file contained in each release artifact for applicable copyright attribution notices.</p><h3><a id="Verify"></a>Verify</h3><p>While recommended, verify is an optional step. You can verify the integrity of any downloaded files using the PGP signatures. Please read <a href="http://httpd.apache.org/dev/verification.html">Verifying Apache HTTP Server Releases</a> for more information on why you should verify our releases.</p><p>The PGP signatures can be verified using PGP or GPG. First download the KEYS file as well as the .asc signature files for the relevant release packages. Make sure you get these files from the main distribution directory linked above, rather than from a mirror. Then verify the signatures using one of the methods below.</p>
 <pre><code>% pgpk -a KEYS
-% pgpv knox-incubating-0.4.0.zip.asc
+% pgpv knox-0.4.0.zip.asc
 </code></pre><p>or</p>
 <pre><code>% pgp -ka KEYS
-% pgp knox-incubating-0.4.0.zip.asc
+% pgp knox-0.4.0.zip.asc
 </code></pre><p>or</p>
 <pre><code>% gpg --import KEYS
-% gpg --verify knox-incubating-0.4.0.zip.asc
+% gpg --verify knox-0.4.0.zip.asc
 </code></pre><h3><a id="4+-+Start+Hadoop+virtual+machine"></a>4 - Start Hadoop virtual machine</h3><p>Start the Hadoop virtual machine.</p><h3><a id="5+-+Install+Knox"></a>5 - Install Knox</h3><p>The steps required to install the gateway will vary depending upon which distribution format (zip | rpm) was downloaded. In either case you will end up with a directory where the gateway is installed. This directory will be referred to as your <code>{GATEWAY_HOME}</code> throughout this document.</p><h4><a id="ZIP"></a>ZIP</h4><p>If you downloaded the Zip distribution you can simply extract the contents into a directory. The example below provides a command that can be executed to do this. Note the <code>{VERSION}</code> portion of the command must be replaced with an actual Apache Knox Gateway version number. This might be 0.4.0 for example and must patch the value in the file downloaded.</p>
-<pre><code>jar xf knox-incubating-{VERSION}.zip
-</code></pre><p>This will create a directory <code>knox-incubating-{VERSION}</code> in your current directory. The directory <code>knox-incubating-{VERSION}</code> will considered your <code>{GATEWAY_HOME}</code></p><h4><a id="RPM"></a>RPM</h4><p>If you downloaded the RPM distribution you can install it using normal RPM package tools. It is important that the user that will be running the gateway server is used to install. This is because several directories are created that are owned by this user. These command will install Knox to <code>/usr/lib/knox</code> following the pattern of other Hadoop components. This directory will be considered your <code>{GATEWAY_HOME}</code>.</p>
-<pre><code>sudo yum localinstall knox-incubating-{VERSION}.rpm
-</code></pre><p>or</p>
-<pre><code>sudo rpm -ihv knox-incubating-{VERSION}.rpm
-</code></pre><h3><a id="6+-+Start+LDAP+embedded+in+Knox"></a>6 - Start LDAP embedded in Knox</h3><p>Knox comes with an LDAP server for demonstration purposes.</p>
+<pre><code>jar xf knox-{VERSION}.zip
+</code></pre><p>This will create a directory <code>knox-{VERSION}</code> in your current directory. The directory <code>knox-{VERSION}</code> will considered your <code>{GATEWAY_HOME}</code></p><h3><a id="6+-+Start+LDAP+embedded+in+Knox"></a>6 - Start LDAP embedded in Knox</h3><p>Knox comes with an LDAP server for demonstration purposes.</p>
 <pre><code>cd {GATEWAY_HOME}
 bin/ldap.sh start
-</code></pre><h3><a id="7+-+Start+Knox"></a>7 - Start Knox</h3><p>The gateway can be started in one of two ways, as java -jar or with a shell script.</p><h6><a id="Starting+via+script"></a>Starting via script</h6><p>Run the knoxcli create-master command in order to persist the master secret that is used to protect the key and credential stores for the gateway instance.</p><h6><a id="linux"></a>linux</h6>
+</code></pre><h3><a id="7+-+Start+Knox"></a>7 - Start Knox</h3><p>The gateway can be started using the provided shell script.</p><h6><a id="Starting+via+script"></a>Starting via script</h6><p>Run the knoxcli create-master command in order to persist the master secret that is used to protect the key and credential stores for the gateway instance.</p><h6><a id="linux"></a>linux</h6>
 <pre><code>cd {GATEWAY_HOME}
 bin/knoxcli.sh create-master
 </code></pre><p>The cli will prompt you for the master secret (i.e. password).</p><p>The server will discover the persisted master secret during start up and complete the setup process for demo installs. A demo install will consist of a knox gateway instance with an identity certificate for localhost. This will require clients to be on the same machine or to turn off hostname verification. For more involved deployments, See the Knox CLI section of this document for additional commands - incuding the ability to create a self-signed certificate for a specific hostname.</p>
@@ -109,7 +103,7 @@ bin/gateway.sh stop
 </code></pre><p>If for some reason the gateway is stopped other than by using the command above you may need to clear the tracking PID.</p>
 <pre><code>cd {GATEWAY_HOME}
 bin/gateway.sh clean
-</code></pre><p><strong>NOTE: This command will also clear any log output in /var/log/knox so use this with caution.</strong></p><h3><a id="8+-+Do+Hadoop+with+Knox"></a>8 - Do Hadoop with Knox</h3><h4><a id="Put+a+file+in+HDFS+via+Knox."></a>Put a file in HDFS via Knox.</h4><h4><a id="CAT+a+file+in+HDFS+via+Knox."></a>CAT a file in HDFS via Knox.</h4><h4><a id="Invoke+the+LISTSATUS+operation+on+WebHDFS+via+the+gateway."></a>Invoke the LISTSATUS operation on WebHDFS via the gateway.</h4><p>This will return a directory listing of the root (i.e. /) directory of HDFS.</p>
+</code></pre><p><strong>NOTE: This command will also clear any .out and .err file from the /var/log/knox directory so use this with caution.</strong></p><h3><a id="8+-+Do+Hadoop+with+Knox"></a>8 - Do Hadoop with Knox</h3><h4><a id="Put+a+file+in+HDFS+via+Knox."></a>Put a file in HDFS via Knox.</h4><h4><a id="CAT+a+file+in+HDFS+via+Knox."></a>CAT a file in HDFS via Knox.</h4><h4><a id="Invoke+the+LISTSTATUS+operation+on+WebHDFS+via+the+gateway."></a>Invoke the LISTSTATUS operation on WebHDFS via the gateway.</h4><p>This will return a directory listing of the root (i.e. /) directory of HDFS.</p>
 <pre><code>curl -i -k -u guest:guest-password -X GET \
     &#39;https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS&#39;
 </code></pre><p>The results of the above command should result in something to along the lines of the output below. The exact information returned is subject to the content within HDFS in your Hadoop cluster. Successfully executing this command at a minimum proves that the gateway is properly configured to provide access to WebHDFS. It does not necessarily provide that any of the other services are correct configured to be accessible. To validate that see the sections for the individual services in <a href="#Service+Details">Service Details</a>.</p>
@@ -124,15 +118,11 @@ Server: Jetty(6.1.26)
 {&quot;accessTime&quot;:0,&quot;blockSize&quot;:0,&quot;group&quot;:&quot;hdfs&quot;,&quot;length&quot;:0,&quot;modificationTime&quot;:1350596040075,&quot;owner&quot;:&quot;hdfs&quot;,&quot;pathSuffix&quot;:&quot;tmp&quot;,&quot;permission&quot;:&quot;777&quot;,&quot;replication&quot;:0,&quot;type&quot;:&quot;DIRECTORY&quot;},
 {&quot;accessTime&quot;:0,&quot;blockSize&quot;:0,&quot;group&quot;:&quot;hdfs&quot;,&quot;length&quot;:0,&quot;modificationTime&quot;:1350595857178,&quot;owner&quot;:&quot;hdfs&quot;,&quot;pathSuffix&quot;:&quot;user&quot;,&quot;permission&quot;:&quot;755&quot;,&quot;replication&quot;:0,&quot;type&quot;:&quot;DIRECTORY&quot;}
 ]}}
-</code></pre><h4><a id="Submit+a+MR+job+via+Knox."></a>Submit a MR job via Knox.</h4><h4><a id="Get+status+of+a+MR+job+via+Knox."></a>Get status of a MR job via Knox.</h4><h4><a id="Cancel+a+MR+job+via+Knox."></a>Cancel a MR job via Knox.</h4><h3><a id="More+Examples"></a>More Examples</h3><h2><a id="Apache+Knox+Details"></a>Apache Knox Details</h2><p>This section provides everything you need to know to get the Knox gateway up and running against a Hadoop cluster.</p><h4><a id="Hadoop"></a>Hadoop</h4><p>An an existing Hadoop 1.x or 2.x cluster is required for Knox sit in front of and protect. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here. It is also possible to use a limited set of services in Hadoop cluster secured with Kerberos. This too required additional configuration that is not described here. See <a href="#Supported+Services">Supported Services</a> for details on what is supported for this release.</p><
 p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow assume a few things:</p>
+</code></pre><h4><a id="Submit+a+MR+job+via+Knox."></a>Submit a MR job via Knox.</h4><h4><a id="Get+status+of+a+MR+job+via+Knox."></a>Get status of a MR job via Knox.</h4><h4><a id="Cancel+a+MR+job+via+Knox."></a>Cancel a MR job via Knox.</h4><h3><a id="More+Examples"></a>More Examples</h3><h2><a id="Apache+Knox+Details"></a>Apache Knox Details</h2><p>This section provides everything you need to know to get the Knox gateway up and running against a Hadoop cluster.</p><h4><a id="Hadoop"></a>Hadoop</h4><p>An existing Hadoop 2.x cluster is required for Knox 0.4.0 to sit in front of and protect. It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration not covered here. It is also possible to protect access to a services of a Hadoop cluster that is secured with kerberos. This too requires additional configuration that is described in other sections of this guide. See <a href="#Supported+Services">Supported Services</a> for details on what is s
 upported for this release.</p><p>The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and Oozie configured, deployed and running. HBase/Stargate and Hive can also be accessed via the Knox Gateway given the proper versions and configuration.</p><p>The instructions that follow assume a few things:</p>
 <ol>
   <li>The gateway is <em>not</em> collocated with the Hadoop clusters themselves.</li>
   <li>The host names and IP addresses of the cluster services are accessible by the gateway where ever it happens to be running.</li>
-</ol><p>All of the instructions and samples provided here are tailored and tested to work &ldquo;out of the box&rdquo; against a <a href="http://hortonworks.com/products/hortonworks-sandbox">Hortonworks Sandbox 2.x VM</a>.</p><h4><a id="Apache+Knox+Directory+Layout"></a>Apache Knox Directory Layout</h4><p>Knox can be installed by expanding the zip file or with rpm. With rpm based install the following directories are created in addition to those described in this section.</p>
-<pre><code>/usr/lib/knox
-/var/log/knox
-/var/run/knox
-</code></pre><p>The directory <code>/usr/lib/knox</code> is considered your <code>{GATEWAY_HOME}</code> and will adhere to the layout described below. The directory <code>/var/log/knox</code> will contain the output files from the server. The directory <code>/var/run/knox</code> will contain the process ID for a currently running gateway server.</p><p>Regardless of the installation method used the layout and content of the <code>{GATEWAY_HOME}</code> will be identical. The table below provides a brief explanation of the important files and directories within <code>{GATEWWAY_HOME}</code></p>
+</ol><p>All of the instructions and samples provided here are tailored and tested to work &ldquo;out of the box&rdquo; against a <a href="http://hortonworks.com/products/hortonworks-sandbox">Hortonworks Sandbox 2.x VM</a>.</p><h4><a id="Apache+Knox+Directory+Layout"></a>Apache Knox Directory Layout</h4><p>Knox can be installed by expanding the zip/archive file.</p><p>The table below provides a brief explanation of the important files and directories within <code>{GATEWWAY_HOME}</code></p>
 <table>
   <thead>
     <tr>
@@ -146,59 +136,79 @@ Server: Jetty(6.1.26)
       <td>Contains configuration files that apply to the gateway globally (i.e. not cluster specific ). </td>
     </tr>
     <tr>
+      <td>data/ </td>
+      <td>Contains security and topology specific artifacts that require read/write access at runtime </td>
+    </tr>
+    <tr>
+      <td>data/topologies/</td>
+      <td>Contains topology files that represent Hadoop clusters which the gateway uses to deploy cluster proxies</td>
+    </tr>
+    <tr>
+      <td>data/security/ </td>
+      <td>Contains the persisted master secret and keystore dir</td>
+    </tr>
+    <tr>
+      <td>data/security/keystores/</td>
+      <td>Contains the gateway identity keystore and credential stores for the gateway and each deployed cluster topology</td>
+    </tr>
+    <tr>
       <td>bin/ </td>
-      <td>Contains the executable shell scripts, batch files and JARs for clients and servers. </td>
+      <td>Contains the executable shell scripts, batch files and JARs for clients and servers.</td>
     </tr>
     <tr>
-      <td>deployments/ </td>
-      <td>Contains topology descriptors used to configure the gateway for specific Hadoop clusters. </td>
+      <td>data/deployments/ </td>
+      <td>Contains deployed cluster topologies used to protect access to specific Hadoop clusters.</td>
     </tr>
     <tr>
       <td>lib/ </td>
-      <td>Contains the JARs for all the components that make up the gateway. </td>
+      <td>Contains the JARs for all the components that make up the gateway.</td>
     </tr>
     <tr>
       <td>dep/ </td>
-      <td>Contains the JARs for all of the components upon which the gateway depends. </td>
+      <td>Contains the JARs for all of the components upon which the gateway depends.</td>
     </tr>
     <tr>
       <td>ext/ </td>
-      <td>A directory where user supplied extension JARs can be placed to extends the gateways functionality. </td>
+      <td>A directory where user supplied extension JARs can be placed to extends the gateways functionality.</td>
+    </tr>
+    <tr>
+      <td>pids/ </td>
+      <td>Contains the process ids for running ldap and gateway servers</td>
     </tr>
     <tr>
       <td>samples/ </td>
-      <td>Contains a number of samples that can be used to explore the functionality of the gateway. </td>
+      <td>Contains a number of samples that can be used to explore the functionality of the gateway.</td>
     </tr>
     <tr>
       <td>templates/ </td>
-      <td>Contains default configuration files that can be copied and customized. </td>
+      <td>Contains default configuration files that can be copied and customized.</td>
     </tr>
     <tr>
       <td>README </td>
-      <td>Provides basic information about the Apache Knox Gateway. </td>
+      <td>Provides basic information about the Apache Knox Gateway.</td>
     </tr>
     <tr>
       <td>ISSUES </td>
-      <td>Describes significant know issues. </td>
+      <td>Describes significant know issues.</td>
     </tr>
     <tr>
       <td>CHANGES </td>
-      <td>Enumerates the changes between releases. </td>
+      <td>Enumerates the changes between releases.</td>
     </tr>
     <tr>
       <td>LICENSE </td>
-      <td>Documents the license under which this software is provided. </td>
+      <td>Documents the license under which this software is provided.</td>
     </tr>
     <tr>
       <td>NOTICE </td>
-      <td>Documents required attribution notices for included dependencies. </td>
+      <td>Documents required attribution notices for included dependencies.</td>
     </tr>
     <tr>
       <td>DISCLAIMER </td>
-      <td>Documents that this release is from a project undergoing incubation at Apache. </td>
+      <td>Documents that this release is from a project undergoing incubation at Apache.</td>
     </tr>
   </tbody>
-</table><h3><a id="Supported+Services"></a>Supported Services</h3><p>This table enumerates the versions of various Hadoop services that have been tested to work with the Knox Gateway. Only more recent versions of some Hadoop components when secured via Kerberos can be accessed via the Knox Gateway.</p>
+</table><h3><a id="Supported+Services"></a>Supported Services</h3><p>This table enumerates the versions of various Hadoop services that have been tested to work with the Knox Gateway.</p>
 <table>
   <thead>
     <tr>
@@ -211,15 +221,15 @@ Server: Jetty(6.1.26)
   <tbody>
     <tr>
       <td>WebHDFS </td>
-      <td>2.1.0 </td>
+      <td>2.4.0 </td>
       <td><img src="check.png"  alt="y"/> </td>
       <td><img src="check.png"  alt="y"/> </td>
     </tr>
     <tr>
       <td>WebHCat/Templeton </td>
-      <td>0.11.0 </td>
+      <td>0.13.0 </td>
+      <td><img src="check.png"  alt="y"/> </td>
       <td><img src="check.png"  alt="y"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
     </tr>
     <tr>
       <td> </td>
@@ -235,45 +245,21 @@ Server: Jetty(6.1.26)
     </tr>
     <tr>
       <td>HBase/Stargate </td>
-      <td>0.95.2 </td>
+      <td>0.98.0 </td>
       <td><img src="check.png"  alt="y"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
-    </tr>
-    <tr>
-      <td>Hive (via WebHCat) </td>
-      <td>0.11.0 </td>
       <td><img src="check.png"  alt="y"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
     </tr>
     <tr>
-      <td> </td>
-      <td>0.12.0 </td>
+      <td>Hive (via WebHCat) </td>
+      <td>0.13.0 </td>
       <td><img src="check.png"  alt="y"/> </td>
       <td><img src="check.png"  alt="y"/> </td>
     </tr>
     <tr>
       <td>Hive (via JDBC) </td>
-      <td>0.11.0 </td>
-      <td><img src="error.png"  alt="n"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td>0.12.0 </td>
+      <td>0.13.0 </td>
+      <td><img src="check.png"  alt="y"/> </td>
       <td><img src="check.png"  alt="y"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
-    </tr>
-    <tr>
-      <td>Hive (via ODBC) </td>
-      <td>0.11.0 </td>
-      <td><img src="error.png"  alt="n"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td>0.12.0 </td>
-      <td><img src="error.png"  alt="n"/> </td>
-      <td><img src="error.png"  alt="n"/> </td>
     </tr>
   </tbody>
 </table><h3><a id="More+Examples"></a>More Examples</h3><p>These examples provide more detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
@@ -291,7 +277,7 @@ Server: Jetty(6.1.26)
 </ul><h3><a id="URL+Mapping"></a>URL Mapping</h3><p>The gateway functions much like a reverse proxy. As such, it maintains a mapping of URLs that are exposed externally by the gateway to URLs that are provided by the Hadoop cluster.</p><h4><a id="Default+Topology+URLs"></a>Default Topology URLs</h4><p>In order to provide compatibility with the Hadoop java client and existing CLI tools, the Knox Gateway has provided a feature called the Default Topology. This refers to a topology deployment that will be able to route URLs without the additional context that the gateway uses for differentiating from one Hadoop cluster to another. This allows the URLs to match those used by existing clients for that may access webhdfs through the Hadoop file system abstraction.</p><p>When a topology file is deployed with a file name that matches the configured default topology name, a specialized mapping for URLs is installed for that particular topology. This allows the URLs that are expected by the e
 xisting Hadoop CLIs for webhdfs to be used in interacting with the specific Hadoop cluster that is represented by the default topology file.</p><p>The configuration for the default topology name is found in gateway-site.xml as a property called: &ldquo;default.app.topology.name&rdquo;.</p><p>The default value for this property is &ldquo;sandbox&rdquo;.</p><p>Therefore, when deploying the sandbox.xml topology, both of the following example URLs work for the same underlying Hadoop cluster:</p>
 <pre><code>https://{gateway-host}:{gateway-port}/webhdfs
 https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/webhdfs
-</code></pre><p>These default topology URLs exist for all of the services in the topology.</p><h4><a id="Fully+Qualified+URLs"></a>Fully Qualified URLs</h4><p>Examples of mappings for the WebHDFS, WebHCat, Oozie and Stargate/HBase are shown below. These mapping are generated from the combination of the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>) and the cluster topology descriptors (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>). The port numbers show for the Cluster URLs represent the default ports for these services. The actual port number may be different for a given cluster.</p>
+</code></pre><p>These default topology URLs exist for all of the services in the topology.</p><h4><a id="Fully+Qualified+URLs"></a>Fully Qualified URLs</h4><p>Examples of mappings for the WebHDFS, WebHCat, Oozie and Stargate/HBase are shown below. These mapping are generated from the combination of the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>) and the cluster topology descriptors (e.g. <code>{GATEWAY_HOME}/conf/topologies/{cluster-name}.xml</code>). The port numbers show for the Cluster URLs represent the default ports for these services. The actual port number may be different for a given cluster.</p>
 <ul>
   <li>WebHDFS
   <ul>
@@ -315,10 +301,10 @@ https://{gateway-host}:{gateway-port}/{g
   </ul></li>
   <li>Hive JDBC
   <ul>
-    <li>Gateway: <code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
+    <li>Gateway: <code>jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
     <li>Cluster: <code>http://{hive-host}:10001/cliservice</code></li>
   </ul></li>
-</ul><p>The values for <code>{gateway-host}</code>, <code>{gateway-port}</code>, <code>{gateway-path}</code> are provided via the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>).</p><p>The value for <code>{cluster-name}</code> is derived from the file name of the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>The value for <code>{webhdfs-host}</code>, <code>{webhcat-host}</code>, <code>{oozie-host}</code>, <code>{hbase-host}</code> and <code>{hive-host}</code> are provided via the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>Note: The ports 50070, 50111, 11000, 60080 and 10001 are the defaults for WebHDFS, WebHCat, Oozie, Stargate/HBase and Hive respectively. Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.</p><h3><a id="Configuration"></a>Configuration</h3><h4><a id="Topolo
 gy+Descriptors"></a>Topology Descriptors</h4><p>The topology descriptor files provide the gateway with per-cluster configuration information. This includes configuration for both the providers within the gateway and the services within the Hadoop cluster. These files are located in <code>{GATEWAY_HOME}/deployments</code>. The general outline of this document looks like this.</p>
+</ul><p>The values for <code>{gateway-host}</code>, <code>{gateway-port}</code>, <code>{gateway-path}</code> are provided via the gateway configuration file (i.e. <code>{GATEWAY_HOME}/conf/gateway-site.xml</code>).</p><p>The value for <code>{cluster-name}</code> is derived from the file name of the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>The value for <code>{webhdfs-host}</code>, <code>{webhcat-host}</code>, <code>{oozie-host}</code>, <code>{hbase-host}</code> and <code>{hive-host}</code> are provided via the cluster topology descriptor (e.g. <code>{GATEWAY_HOME}/deployments/{cluster-name}.xml</code>).</p><p>Note: The ports 50070, 50111, 11000, 60080 (default 8080) and 10001 are the defaults for WebHDFS, WebHCat, Oozie, Stargate/HBase and Hive respectively. Their values can also be provided via the cluster topology descriptor if your Hadoop cluster uses different ports.</p><h3><a id="Configuration"></a>Configuration</h3><h
 4><a id="Topology+Descriptors"></a>Topology Descriptors</h4><p>The topology descriptor files provide the gateway with per-cluster configuration information. This includes configuration for both the providers within the gateway and the services within the Hadoop cluster. These files are located in <code>{GATEWAY_HOME}/deployments</code>. The general outline of this document looks like this.</p>
 <pre><code>&lt;topology&gt;
     &lt;gateway&gt;
         &lt;provider&gt;
@@ -348,7 +334,7 @@ https://{gateway-host}:{gateway-port}/{g
 &lt;/service&gt;
 </code></pre>
 <dl><dt>/topology/service</dt><dd>Provider information about a particular service within the Hadoop cluster. Not all services are necessarily exposed as gateway endpoints.</dd><dt>/topology/service/role</dt><dd>Identifies the role of this service. Currently supported roles are: WEBHDFS, WEBHCAT, WEBHBASE, OOZIE, HIVE, NAMENODE, JOBTRACKER Additional service roles can be supported via plugins.</dd><dt>topology/service/url</dt><dd>The URL identifying the location of a particular service within the Hadoop cluster.</dd>
-</dl><h4><a id="Hostmap+Provider"></a>Hostmap Provider</h4><p>The purpose of the Hostmap provider is to handle situations where host are know by one name within the cluster and another name externally. This frequently occurs when virtual machines are used and in particular using cloud hosting services. Currently the Hostmap provider is configured as part of the topology file. The basic structure is shown below.</p>
+</dl><h4><a id="Hostmap+Provider"></a>Hostmap Provider</h4><p>The purpose of the Hostmap provider is to handle situations where host are known by one name within the cluster and another name externally. This frequently occurs when virtual machines are used and in particular when using cloud hosting services. Currently, the Hostmap provider is configured as part of the topology file. The basic structure is shown below.</p>
 <pre><code>&lt;topology&gt;
     &lt;gateway&gt;
         ...
@@ -391,7 +377,7 @@ ip-10-39-107-209.ec2.internal
     &lt;/gateway&gt;
     ...
 &lt;/topology&gt;
-</code></pre><h5><a id="Hostmap+Provider+Example+-+Sandbox"></a>Hostmap Provider Example - Sandbox</h5><p>Hortonwork&rsquo;s Sandbox 2.x poses a different challenge for host name mapping. This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it is accessible via localhost. However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the host name. So from the perspective of a client accessing Sandbox the external host name is localhost. The Hostmap configuration required to allow access to Sandbox from the host operating system is this.</p>
+</code></pre><h5><a id="Hostmap+Provider+Example+-+Sandbox"></a>Hostmap Provider Example - Sandbox</h5><p>The Hortonworks Sandbox 2.x poses a different challenge for host name mapping. This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it is accessible via localhost. However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the host name. So from the perspective of a client accessing Sandbox the external host name is localhost. The Hostmap configuration required to allow access to Sandbox from the host operating system is this.</p>
 <pre><code>&lt;topology&gt;
     &lt;gateway&gt;
         ...
@@ -409,19 +395,19 @@ ip-10-39-107-209.ec2.internal
 <dl><dt>topology/gateway/provider/role</dt><dd>The role for a Hostmap provider must always be <code>hostmap</code>.</dd><dt>topology/gateway/provider/name</dt><dd>The Hostmap provider supplied out-of-the-box is selected via the name <code>static</code>.</dd><dt>topology/gateway/provider/enabled</dt><dd>Host mapping can be enabled or disabled by providing <code>true</code> or <code>false</code>.</dd><dt>topology/gateway/provider/param</dt><dd>Host mapping is configured by providing parameters for each external to internal mapping.</dd><dt>topology/gateway/provider/param/name</dt><dd>The parameter names represent an external host names associated with the internal host names provided by the value element. This can be a comma separated list of host names that all represent the same physical host. When mapping from internal to external host name the first external host name in the list is used.</dd><dt>topology/gateway/provider/param/value</dt><dd>The parameter values represent the inte
 rnal host names associated with the external host names provider by the name element. This can be a comma separated list of host names that all represent the same physical host. When mapping from external to internal host names the first internal host name in the list is used.</dd>
 </dl><h4><a id="Logging"></a>Logging</h4><p>If necessary you can enable additional logging by editing the <code>log4j.properties</code> file in the <code>conf</code> directory. Changing the rootLogger value from <code>ERROR</code> to <code>DEBUG</code> will generate a large amount of debug logging. A number of useful, more fine loggers are also provided in the file.</p><h4><a id="Java+VM+Options"></a>Java VM Options</h4><p>TODO - Java VM options doc.</p><h4><a id="Persisting+the+Master+Secret"></a>Persisting the Master Secret</h4><p>The master secret is required to start the server. This secret is used to access secured artifacts by the gateway instance. Keystore, trust stores and credential stores are all protected with the master secret.</p><p>You may persist the master secret by supplying the <em>-persist-master</em> switch at startup. This will result in a warning indicating that persisting the secret is less secure than providing it at startup. We do make some provisions in ord
 er to protect the persisted password.</p><p>It is encrypted with AES 128 bit encryption and where possible the file permissions are set to only be accessible by the user that the gateway is running as.</p><p>After persisting the secret, ensure that the file at config/security/master has the appropriate permissions set for your environment. This is probably the most important layer of defense for master secret. Do not assume that the encryption if sufficient protection.</p><p>A specific user should be created to run the gateway this user will be the only user with permissions for the persisted master file.</p><p>See the Knox CLI section for descriptions of the command line utilties related to the master secret.</p><h4><a id="Management+of+Security+Artifacts"></a>Management of Security Artifacts</h4><p>There are a number of artifacts that are used by the gateway in ensuring the security of wire level communications, access to protected resources and the encryption of sensitive data. T
 hese artifacts can be managed from outside of the gateway instances or generated and populated by the gateway instance itself.</p><p>The following is a description of how this is coordinated with both standalone (development, demo, etc) gateway instances and instances as part of a cluster of gateways in mind.</p><p>Upon start of the gateway server we:</p>
 <ol>
-  <li>Look for an identity store at <code>conf/security/keystores/gateway.jks</code>.  The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
+  <li>Look for an identity store at <code>data/security/keystores/gateway.jks</code>.  The identity store contains the certificate and private key used to represent the identity of the server for SSL connections and signature creation.
   <ul>
     <li>If there is no identity store we create one and generate a self-signed certificate for use in standalone/demo mode.  The certificate is stored with an alias of gateway-identity.</li>
-    <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias with called gateway-identity.</li>
+    <li>If there is an identity store found than we ensure that it can be loaded using the provided master secret and that there is an alias called gateway-identity.</li>
   </ul></li>
-  <li>Look for a credential store at <code>conf/security/keystores/__gateway-credentials.jceks</code>.  This credential store is used to store secrets/passwords that are used by the gateway.  For instance, this is where the pass-phrase for accessing the gateway-identity certificate is kept.
+  <li>Look for a credential store at <code>data/security/keystores/__gateway-credentials.jceks</code>.  This credential store is used to store secrets/passwords that are used by the gateway.  For instance, this is where the passphrase for accessing the gateway-identity certificate is kept.
   <ul>
-    <li>If there is no credential store found then we create one and populate it with a generated pass-phrase for the alias <code>gateway-identity-passphrase</code>.  This is coordinated with the population of the self-signed cert into the identity-store.</li>
+    <li>If there is no credential store found then we create one and populate it with a generated passphrase for the alias <code>gateway-identity-passphrase</code>.  This is coordinated with the population of the self-signed cert into the identity-store.</li>
     <li>If a credential store is found then we ensure that it can be loaded using the provided master secret and that the expected aliases have been populated with secrets.</li>
   </ul></li>
 </ol><p>Upon deployment of a Hadoop cluster topology within the gateway we:</p>
 <ol>
-  <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>conf/security/keystores/sandbox-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.
+  <li>Look for a credential store for the topology. For instance, we have a sample topology that gets deployed out of the box. We look for <code>data/security/keystores/sandbox-credentials.jceks</code>. This topology specific credential store is used for storing secrets/passwords that are used for encrypting sensitive data with topology specific keys.
   <ul>
     <li>If no credential store is found for the topology being deployed then one is created for it.  Population of the aliases is delegated to the configured providers within the system that will require the use of a secret for a particular task.  They may programmatic set the value of the secret or choose to have the value for the specified alias generated through the AliasService.</li>
     <li>If a credential store is found then we ensure that it can be loaded with the provided master secret and the configured providers have the opportunity to ensure that the aliases are populated and if not to populate them.</li>
@@ -431,7 +417,7 @@ ip-10-39-107-209.ec2.internal
   <li>Using a single gateway instance as a master instance the artifacts can be generated or placed into the expected location and then replicated across all of the slave instances before startup.</li>
   <li>Using an NFS mount as a central location for the artifacts would provide a single source of truth without the need to replicate them over the network. Of course, NFS mounts have their own challenges.</li>
   <li>Using the KnoxCLI to create and manage the security artifacts.</li>
-</ol><p>See the Knox CLI section for descriptions of the command line utilties related to the security artifact management.</p><h4><a id="Keystores"></a>Keystores</h4><p>In order to provide your own certificate for use by the gateway, you will need to either import an existing key pair into a Java keystore or generate a self-signed cert using the Java keytool.</p><h5><a id="Importing+a+key+pair+into+a+Java+keystore"></a>Importing a key pair into a Java keystore</h5><h1><a id="----NEEDS+TESTING"></a>&mdash;-NEEDS TESTING</h1><p>One way to accomplish this is to start with a PKCS12 store for your key pair and then convert it to a Java keystore or JKS.</p>
+</ol><p>See the Knox CLI section for descriptions of the command line utilties related to the security artifact management.</p><h4><a id="Keystores"></a>Keystores</h4><p>In order to provide your own certificate for use by the gateway, you will need to either import an existing key pair into a Java keystore or generate a self-signed cert using the Java keytool.</p><h5><a id="Importing+a+key+pair+into+a+Java+keystore"></a>Importing a key pair into a Java keystore</h5><p>One way to accomplish this is to start with a PKCS12 store for your key pair and then convert it to a Java keystore or JKS.</p>
 <pre><code>openssl pkcs12 -export -in cert.pem -inkey key.pem &gt; server.p12
 </code></pre><p>The above example uses openssl to create a PKCS12 encoded store for your provided certificate private key.</p>
 <pre><code>keytool -importkeystore -srckeystore {server.p12} -destkeystore gateway.jks -srcstoretype pkcs12
@@ -439,17 +425,17 @@ ip-10-39-107-209.ec2.internal
 <ol>
   <li>the alias MUST be &ldquo;gateway-identity&rdquo;</li>
   <li>the name of the expected identity keystore for the gateway MUST be gateway.jks</li>
-  <li>the passwords for the keystore and the imported key MUST both be the master secret for the gateway install</li>
-</ol><p>NOTE: The password for the keystore as well as that of the imported key must be the master secret for the gateway instance.</p><h1><a id="----END+NEEDS+TESTING"></a>&mdash;-END NEEDS TESTING</h1><h5><a id="Generating+a+self-signed+cert+for+use+in+testing+or+development+environments"></a>Generating a self-signed cert for use in testing or development environments</h5>
+  <li>the passwords for the keystore and the imported key may both be set to the master secret for the gateway install</li>
+</ol><p>NOTE: The password for the keystore as well as that of the imported key may be the master secret for the gateway instance or you may set the gateway-identity-passphrase alias using the Knox CLI to the actual key passphrase. See the Knox CLI section for details.</p><h5><a id="Generating+a+self-signed+cert+for+use+in+testing+or+development+environments"></a>Generating a self-signed cert for use in testing or development environments</h5>
 <pre><code>keytool -genkey -keyalg RSA -alias gateway-identity -keystore gateway.jks \
     -storepass {master-secret} -validity 360 -keysize 2048
-</code></pre><p>Keytool will prompt you for a number of elements used will comprise the distiniguished name (DN) within your certificate. </p><p><em>NOTE:</em> When it prompts you for your First and Last name be sure to type in the hostname of the machine that your gateway instance will be running on. This is used by clients during hostname verification to ensure that the presented certificate matches the hostname that was used in the URL for the connection - so they need to match.</p><p><em>NOTE:</em> When it prompts for the key password just press enter to ensure that it is the same as the keystore password. Which as was described earlier must match the master secret for the gateway instance.</p><p>See the Knox CLI section for descriptions of the command line utilties related to the management of the keystores.</p><h5><a id="Credential+Store"></a>Credential Store</h5><p>Whenever you provide your own keystore with either a self-signed cert or a real certificate signed by a trusted 
 authority, you will need to create an empty credential store. This is necessary for the current release in order for the system to utilize the same password for the keystore and the key.</p><p>The credential stores in Knox use the JCEKS keystore type as it allows for the storage of general secrets in addition to certificates.</p><p>Keytool may be used to create credential stores but the Knox CLI section details how to create aliases. These aliases are managed within credential stores which are created by the CLI as appropriate. </p><p>See the Knox CLI section for descriptions of the command line utilties related to the management of the credential stores.</p><h5><a id="Provisioning+of+Keystores"></a>Provisioning of Keystores</h5><p>Once you have created these keystores you must move them into place for the gateway to discover them and use them to represent its identity for SSL connections. This is done by copying the keystores to the <code>{GATEWAY_HOME}/conf/security/keystores</cod
 e> directory for your gateway install.</p><h4><a id="Summary+of+Secrets+to+be+Managed"></a>Summary of Secrets to be Managed</h4>
+</code></pre><p>Keytool will prompt you for a number of elements used will comprise the distiniguished name (DN) within your certificate. </p><p><em>NOTE:</em> When it prompts you for your First and Last name be sure to type in the hostname of the machine that your gateway instance will be running on. This is used by clients during hostname verification to ensure that the presented certificate matches the hostname that was used in the URL for the connection - so they need to match.</p><p><em>NOTE:</em> When it prompts for the key password just press enter to ensure that it is the same as the keystore password. Which as was described earlier must match the master secret for the gateway instance. Alternatively, you can set it to another passphrase - take note of it and set the gateway-identity-passphrase alias to that passphrase using the Knox CLI.</p><p>See the Knox CLI section for descriptions of the command line utilties related to the management of the keystores.</p><h5><a id="Cre
 dential+Store"></a>Credential Store</h5><p>Whenever you provide your own keystore with either a self-signed cert or an issued certificate signed by a trusted authority, you will need to set an alias for the gateway-identity-passphrase or create an empty credential store. This is necessary for the current release in order for the system to determine the correct password for the keystore and the key.</p><p>The credential stores in Knox use the JCEKS keystore type as it allows for the storage of general secrets in addition to certificates.</p><p>Keytool may be used to create credential stores but the Knox CLI section details how to create aliases. These aliases are managed within credential stores which are created by the CLI as needed. The simplest approach is to create the gateway-identity-passpharse alias with the Knox CLI. This will create the credential store if it doesn&rsquo;t already exist and add the key passphrase.</p><p>See the Knox CLI section for descriptions of the comman
 d line utilties related to the management of the credential stores.</p><h5><a id="Provisioning+of+Keystores"></a>Provisioning of Keystores</h5><p>Once you have created these keystores you must move them into place for the gateway to discover them and use them to represent its identity for SSL connections. This is done by copying the keystores to the <code>{GATEWAY_HOME}/data/security/keystores</code> directory for your gateway install.</p><h4><a id="Summary+of+Secrets+to+be+Managed"></a>Summary of Secrets to be Managed</h4>
 <ol>
   <li>Master secret - the same for all gateway instances in a cluster of gateways</li>
   <li>All security related artifacts are protected with the master secret</li>
   <li>Secrets used by the gateway itself are stored within the gateway credential store and are the same across all gateway instances in the cluster of gateways</li>
   <li>Secrets used by providers within cluster topologies are stored in topology specific credential stores and are the same for the same topology across the cluster of gateway instances.  However, they are specific to the topology - so secrets for one hadoop cluster are different from those of another.  This allows for fail-over from one gateway instance to another even when encryption is being used while not allowing the compromise of one encryption key to expose the data for all clusters.</li>
-</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as host name verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h3><a id="Knox+CLI"></a>Knox CLI</h3><p>The Knox CLI is a command line utility for management of various aspects of the Knox deployment. It is primarily concerned with the management of the security artifacts for the gateway instance and each of the deployed topologies or hadoop clusters that are gated by the Knox Gateway instance.</p><p>The various security artifacts are also generated and populated automatically by the Knox Gateway runtime when they are not found at startup. The assumptions made in those cases are appropriate for a test or development gateway instance
  and assume &lsquo;localhost&rsquo; for hostname specific activities. For production deployments the use of the CLI may aid in managing some production deployments.</p><p>The knoxcli.sh script is located in the {GATEWAY_HOME}/bin directory.</p><h4><a id="Help"></a>Help</h4><h5><a id="knoxcli.sh+[--help]"></a>knoxcli.sh [&ndash;help]</h5><p>prints help for all commands</p><h4><a id="Master+secret+persistence"></a>Master secret persistence</h4><h5><a id="knoxcli.sh+create-master+[--help]"></a>knoxcli.sh create-master [&ndash;help]</h5><p>Creates and persists an encrypted master secret in a file within {GATEWAY_HOME}/data/security/master. </p><p>NOTE: This command fails when there is an existing master file in the expected location.</p><h4><a id="Alias+creation"></a>Alias creation</h4><h5><a id="knoxcli.sh+create-alias+n+[--cluster+c]+[--value+v]+[--generate]+[--help]"></a>knoxcli.sh create-alias n [&ndash;cluster c] [&ndash;value v] [&ndash;generate] [&ndash;help]</h5><p>Creates a pas
 sword alias and stores it in a credential store within the {GATEWAY_HOME}/data/security/keystores dir. </p>
+</ol><p>NOTE: the SSL certificate will need special consideration depending on the type of certificate. Wildcard certs may be able to be shared across all gateway instances in a cluster. When certs are dedicated to specific machines the gateway identity store will not be able to be blindly replicated as host name verification problems will ensue. Obviously, trust-stores will need to be taken into account as well.</p><h3><a id="Knox+CLI"></a>Knox CLI</h3><p>The Knox CLI is a command line utility for management of various aspects of the Knox deployment. It is primarily concerned with the management of the security artifacts for the gateway instance and each of the deployed topologies or hadoop clusters that are gated by the Knox Gateway instance.</p><p>The various security artifacts are also generated and populated automatically by the Knox Gateway runtime when they are not found at startup. The assumptions made in those cases are appropriate for a test or development gateway instance
  and assume &lsquo;localhost&rsquo; for hostname specific activities. For production deployments the use of the CLI may aid in managing some production deployments.</p><p>The knoxcli.sh script is located in the {GATEWAY_HOME}/bin directory.</p><h4><a id="Help"></a>Help</h4><h5><a id="knoxcli.sh+[--help]"></a>knoxcli.sh [&ndash;help]</h5><p>prints help for all commands</p><h4><a id="Knox+Verison+Info"></a>Knox Verison Info</h4><h5><a id="knoxcli.sh+version+[--help]"></a>knoxcli.sh version [&ndash;help]</h5><p>Displays Knox version information.</p><h4><a id="Master+secret+persistence"></a>Master secret persistence</h4><h5><a id="knoxcli.sh+create-master+[--force][--help]"></a>knoxcli.sh create-master [&ndash;force][&ndash;help]</h5><p>Creates and persists an encrypted master secret in a file within {GATEWAY_HOME}/data/security/master. </p><p>NOTE: This command fails when there is an existing master file in the expected location. You may force it to overwrite the master file with the &
 ndash;force switch. NOTE: this will require you to change passwords protecting the keystores for the gateway identity keystores and all credential stores.</p><h4><a id="Alias+creation"></a>Alias creation</h4><h5><a id="knoxcli.sh+create-alias+n+[--cluster+c]+[--value+v]+[--generate]+[--help]"></a>knoxcli.sh create-alias n [&ndash;cluster c] [&ndash;value v] [&ndash;generate] [&ndash;help]</h5><p>Creates a password alias and stores it in a credential store within the {GATEWAY_HOME}/data/security/keystores dir. </p>
 <table>
   <thead>
     <tr>
@@ -521,7 +507,7 @@ ip-10-39-107-209.ec2.internal
       <td>name of the host to be used in the self-signed certificate. This allows multi-host deployments to specify the proper hostnames for hostname verification to succeed on the client side of the SSL connection. The default is “localhost”.</td>
     </tr>
   </tbody>
-</table><h3><a id="Authentication"></a>Authentication</h3><p>There are two types of providers supported in Knox for establishing a user&rsquo;s identity:</p>
+</table><h4><a id="Topology+Redeploy"></a>Topology Redeploy</h4><h4><a id="redeploy+[--cluster+c]"></a>redeploy [&ndash;cluster c]</h4><p>Redeploys one or all of the gateway&rsquo;s clusters (a.k.a topologies).</p><h3><a id="Authentication"></a>Authentication</h3><p>There are two types of providers supported in Knox for establishing a user&rsquo;s identity:</p>
 <ol>
   <li>Authentication Providers</li>
   <li>Federation Providers</li>
@@ -1013,7 +999,7 @@ exit
     &lt;name&gt;webhcat.proxyuser.knox.hosts&lt;/name&gt;
     &lt;value&gt;FQDN_OF_KNOX_HOST&lt;/value&gt;
 &lt;/property&gt;
-</code></pre><h4><a id="Grant+proxy+privilege+for+Knox+in+`webhcat-stie.xml`+on+Hadoop+master+nodes"></a>Grant proxy privilege for Knox in <code>webhcat-stie.xml</code> on Hadoop master nodes</h4><p>Update <code>webhcat-site.xml</code> and add the following lines towards the end of the file.</p><p>Replace FQDN_OF_KNOX_HOST with right value in your cluster. You could use * for local developer testing if Knox host does not have static IP.</p>
+</code></pre><h4><a id="Grant+proxy+privilege+for+Knox+in+`webhcat-site.xml`+on+Hadoop+master+nodes"></a>Grant proxy privilege for Knox in <code>webhcat-site.xml</code> on Hadoop master nodes</h4><p>Update <code>webhcat-site.xml</code> and add the following lines towards the end of the file.</p><p>Replace FQDN_OF_KNOX_HOST with right value in your cluster. You could use * for local developer testing if Knox host does not have static IP.</p>
 <pre><code>&lt;property&gt;
     &lt;name&gt;hadoop.proxyuser.knox.groups&lt;/name&gt;
     &lt;value&gt;users&lt;/value&gt;
@@ -1022,7 +1008,7 @@ exit
     &lt;name&gt;hadoop.proxyuser.knox.hosts&lt;/name&gt;
     &lt;value&gt;FQDN_OF_KNOX_HOST&lt;/value&gt;
 &lt;/property&gt;
-</code></pre><h4><a id="Grant+proxy+privilege+for+Knox+in+`oozie-stie.xml`+on+Oozie+host"></a>Grant proxy privilege for Knox in <code>oozie-stie.xml</code> on Oozie host</h4><p>Update <code>oozie-site.xml</code> and add the following lines towards the end of the file.</p><p>Replace FQDN_OF_KNOX_HOST with right value in your cluster. You could use * for local developer testing if Knox host does not have static IP.</p>
+</code></pre><h4><a id="Grant+proxy+privilege+for+Knox+in+`oozie-site.xml`+on+Oozie+host"></a>Grant proxy privilege for Knox in <code>oozie-site.xml</code> on Oozie host</h4><p>Update <code>oozie-site.xml</code> and add the following lines towards the end of the file.</p><p>Replace FQDN_OF_KNOX_HOST with right value in your cluster. You could use * for local developer testing if Knox host does not have static IP.</p>
 <pre><code>&lt;property&gt;
    &lt;name&gt;oozie.service.ProxyUserService.proxyuser.knox.groups&lt;/name&gt;
    &lt;value&gt;users&lt;/value&gt;
@@ -1031,7 +1017,7 @@ exit
    &lt;name&gt;oozie.service.ProxyUserService.proxyuser.knox.hosts&lt;/name&gt;
    &lt;value&gt;FQDN_OF_KNOX_HOST&lt;/value&gt;
 &lt;/property&gt;
-</code></pre><h4><a id="Copy+knox+keytab+to+Knox+host"></a>Copy knox keytab to Knox host</h4><p>Add unix account for the knox user on Knox host</p>
+</code></pre><h4><a id="Enable+http+transport+mode+and+use+substitution+in+Hive+Server2"></a>Enable http transport mode and use substitution in Hive Server2</h4><p>Update <code>hive-site.xml</code> and set the following properties on Hive Server2 hosts. Some of the properties may already be in the hive-site.xml. Ensure that the values match the ones below.</p><p><property>  <name>hive.server2.allow.user.substitution</name>  <value>true</value> </property></p><p><property>  <name>hive.server2.transport.mode</name>  <value>http</value>  <description>Server transport mode. &ldquo;binary&rdquo; or &ldquo;http&rdquo;.</description> </property></p><p><property>  <name>hive.server2.thrift.http.port</name>  <value>10001</value>  <description>Port number when in HTTP mode.</description> </property></p><p><property>  <name>hive.server2.thrift.http.path</name>  <value>cliservice</value>  <description>Path component of URL endpoint when in HTTP mode.</description> </property></p><h4><a id="Copy
 +knox+keytab+to+Knox+host"></a>Copy knox keytab to Knox host</h4><p>Add unix account for the knox user on Knox host</p>
 <pre><code>useradd -g hadoop knox
 </code></pre><p>Copy knox.service.keytab created on KDC host on to your Knox host /etc/knox/conf/knox.service.keytab</p>
 <pre><code>chown knox knox.service.keytab
@@ -1124,7 +1110,7 @@ APACHE_HOME/bin/apachectl -k stop
   </tbody>
 </table><h4><a id="REST+Invocation"></a>REST Invocation</h4><p>The following curl command can be used to request a directory listing from HDFS while passing in the expected header X-XSRF-Header.</p>
 <pre><code>curl -k -i --header &quot;X-XSRF-Header: valid&quot; -v -u guest:guest-password https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS
-</code></pre><p>Omitting the &ndash;header &ldquo;X-XSRF-Header: valid&rdquo; above should result in an HTTP 400 bad_request.</p><p>Disabling the provider will then allow a request that is missing the header through. </p><h3><a id="Preauthenticated+SSO+Provider"></a>Preauthenticated SSO Provider</h3><p>A number of SSO solutions provide mechanisms for federating an authenticated identity across applications. These mechanisms are at times simple HTTP Header type tokens that can be used to propagate the identity across process boundaries.</p><p>Knox Gateway needs a pluggable mechanism for consuming these tokens and federating the asserted identity through an interaction with the Hadoop cluster. </p><p><strong>CAUTION: The use of this provider requires that proper network security and identity provider configuration and deployment does not allow requests directly to the Knox gateway. Otherwise, this provider will leave the gateway exposed to identity spoofing.</strong></p><h4><a id="Con
 figuration"></a>Configuration</h4><h5><a id="Overview"></a>Overview</h5><p>The HeaderPreAuth provider is configured within the topology file and has a minimal configuration that assumes SM_USER for CA SiteMinder. The following example is the bare minimum configuration for SiteMinder (with no IP address validation).</p>
+</code></pre><p>Omitting the &ndash;header &ldquo;X-XSRF-Header: valid&rdquo; above should result in an HTTP 400 bad_request.</p><p>Disabling the provider will then allow a request that is missing the header through. </p><h3><a id="Preauthenticated+SSO+Provider"></a>Preauthenticated SSO Provider</h3><p>A number of SSO solutions provide mechanisms for federating an authenticated identity across applications. These mechanisms are at times simple HTTP Header type tokens that can be used to propagate the identity across process boundaries.</p><p>Knox Gateway needs a pluggable mechanism for consuming these tokens and federating the asserted identity through an interaction with the Hadoop cluster. </p><p><strong>CAUTION: The use of this provider requires that proper network security and identity provider configuration and deployment does not allow requests directly to the Knox gateway. Otherwise, this provider will leave the gateway exposed to identity spoofing.</strong></p><h4><a id="Con
 figuration"></a>Configuration</h4><h5><a id="Overview"></a>Overview</h5><p>This provider was designed for use with identity solutions such as those provided by CA&rsquo;s SiteMinder and IBM&rsquo;s Tivoli Access Manager. While direct testing with these products has not been done, there has been extensive unit and functional testing that ensure that it should work with such providers.</p><p>The HeaderPreAuth provider is configured within the topology file and has a minimal configuration that assumes SM_USER for CA SiteMinder. The following example is the bare minimum configuration for SiteMinder (with no IP address validation).</p>
 <pre><code>&lt;provider&gt;
   &lt;role&gt;federation&lt;/role&gt;
   &lt;name&gt;HeaderPreAuth&lt;/name&gt;
@@ -1275,7 +1261,7 @@ APACHE_HOME/bin/apachectl -k stop
 <pre><code>set verbosity QUIET
 set show-last-result false
 </code></pre><p>Also when running interactively use the <code>exit</code> command to terminate the shell. Using <code>^C</code> to exit can sometimes leaves the parent shell in a problematic state.</p><p>The shell can also be used to execute a script by passing a single filename argument.</p>
-<pre><code>java -jar bin/shell.jar samples/ExampleWebHdfsPutGetFile.groovy
+<pre><code>java -jar bin/shell.jar samples/ExampleWebHdfsPutGet.groovy
 </code></pre><h3><a id="Examples"></a>Examples</h3><p>Once the shell can be launched the DSL can be used to interact with the gateway and Hadoop. Below is a very simple example of an interactive shell session to upload a file to HDFS.</p>
 <pre><code>java -jar bin/shell.jar
 knox:000&gt; session = Hadoop.login( &quot;https://localhost:8443/gateway/sandbox&quot;, &quot;guest&quot;, &quot;guest-password&quot; )
@@ -1485,16 +1471,16 @@ dep/httpcore-4.2.2.jar
 dep/commons-lang3-3.1.jar
 dep/commons-codec-1.7.jar
 </code></pre><p>So on Linux/MacOS you would need this command</p>
-<pre><code>groovy -cp lib/gateway-shell-0.2.0-SNAPSHOT.jar:dep/httpclient-4.2.3.jar:dep/httpcore-4.2.2.jar:dep/commons-lang3-3.1.jar:dep/commons-codec-1.7.jar samples/ExampleWebHdfsPutGet.groovy
+<pre><code>groovy -cp lib/gateway-shell-0.4.0.jar:dep/httpclient-4.2.5.jar:dep/httpcore-4.2.4.jar:dep/commons-lang3-3.1.jar:dep/commons-codec-1.7.jar samples/ExampleWebHdfsPutGet.groovy
 </code></pre><p>and on Windows you would need this command</p>
-<pre><code>groovy -cp lib/gateway-shell-0.2.0-SNAPSHOT.jar;dep/httpclient-4.2.3.jar;dep/httpcore-4.2.2.jar;dep/commons-lang3-3.1.jar;dep/commons-codec-1.7.jar samples/ExampleWebHdfsPutGet.groovy
-</code></pre><p>The exact list of required JARs is likely to change from release to release so it is recommended that you utilize the wrapper <code>bin/shell.jar</code>.</p><p>In addition because the DSL can be used via standard Groovy, the Groovy integrations in many popular IDEs (e.g. IntelliJ , Eclipse) can also be used. This makes it particularly nice to develop and execute scripts to interact with Hadoop. The code-completion features in modern IDEs in particular provides immense value. All that is required is to add the shell-0.2.0.jar to the projects class path.</p><p>There are a variety of Groovy tools that make it very easy to work with the standard interchange formats (i.e. JSON and XML). In Groovy the creation of XML or JSON is typically done via a &ldquo;builder&rdquo; and parsing done via a &ldquo;slurper&rdquo;. In addition once JSON or XML is &ldquo;slurped&rdquo; the GPath, an XPath like feature build into Groovy can be used to access data.</p>
+<pre><code>groovy -cp lib/gateway-shell-0.4.0.jar;dep/httpclient-4.2.5.jar;dep/httpcore-4.2.4.jar;dep/commons-lang3-3.1.jar;dep/commons-codec-1.7.jar samples/ExampleWebHdfsPutGet.groovy
+</code></pre><p>The exact list of required JARs is likely to change from release to release so it is recommended that you utilize the wrapper <code>bin/shell.jar</code>.</p><p>In addition because the DSL can be used via standard Groovy, the Groovy integrations in many popular IDEs (e.g. IntelliJ , Eclipse) can also be used. This makes it particularly nice to develop and execute scripts to interact with Hadoop. The code-completion features in modern IDEs in particular provides immense value. All that is required is to add the gateway-shell-0.4.0.jar to the projects class path.</p><p>There are a variety of Groovy tools that make it very easy to work with the standard interchange formats (i.e. JSON and XML). In Groovy the creation of XML or JSON is typically done via a &ldquo;builder&rdquo; and parsing done via a &ldquo;slurper&rdquo;. In addition once JSON or XML is &ldquo;slurped&rdquo; the GPath, an XPath like feature build into Groovy can be used to access data.</p>
 <ul>
   <li>XML
   <ul>
-    <li>Markup Builder [Overview]<a href="http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder)">http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder)</a>, <a href="http://groovy.codehaus.org/api/groovy/xml/MarkupBuilder.html">API</a></li>
-    <li>XML Slurper [Overview]<a href="http://groovy.codehaus.org/Reading+XML+using+Groovy's+XmlSlurper)">http://groovy.codehaus.org/Reading+XML+using+Groovy's+XmlSlurper)</a>, <a href="http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html">API</a></li>
-    <li>XPath [Overview]<a href="http://groovy.codehaus.org/GPath)">http://groovy.codehaus.org/GPath)</a>, <a href="http://docs.oracle.com/javase/1.5.0/docs/api/javax/xml/xpath/XPath.html">API</a></li>
+    <li>Markup Builder <a href="http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder">Overview</a>, <a href="http://groovy.codehaus.org/api/groovy/xml/MarkupBuilder.html">API</a></li>
+    <li>XML Slurper <a href="http://groovy.codehaus.org/Reading+XML+using+Groovy's+XmlSlurper">Overview</a>, <a href="http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html">API</a></li>
+    <li>XPath <a href="http://groovy.codehaus.org/GPath">Overview</a>, <a href="http://docs.oracle.com/javase/1.5.0/docs/api/javax/xml/xpath/XPath.html">API</a></li>
   </ul></li>
   <li>JSON
   <ul>
@@ -1939,9 +1925,21 @@ curl -i -k -u guest:guest-password -X DE
   <ul>
     <li><code>Workflow.status(session).jobId(jobId).now().string</code></li>
   </ul></li>
-</ul><h3><a id="HBase"></a>HBase</h3><p>TODO</p><h4><a id="HBase+URL+Mapping"></a>HBase URL Mapping</h4><p>TODO</p><h4><a id="HBase+Examples"></a>HBase Examples</h4><p>TODO</p><p>The examples below illustrate the set of basic operations with HBase instance using Stargate REST API. Use following link to get more more details about HBase/Stargate API: <a href="http://wiki.apache.org/hadoop/Hbase/Stargate">http://wiki.apache.org/hadoop/Hbase/Stargate</a>.</p><p>Note: Some HBase examples may not work due to enabled <a href="https://hbase.apache.org/book/hbase.accesscontrol.configuration.html">Access Control</a>. User may not be granted for performing operations in samples. In order to check if Access Control is configured in the HBase instance verify hbase-site.xml for a presence of <code>org.apache.hadoop.hbase.security.access.AccessController</code> in <code>hbase.coprocessor.master.classes</code> and <code>hbase.coprocessor.region.classes</code> properties.<br/>To grant the Read, Wri
 te, Create permissions to <code>guest</code> user execute the following command:</p>
+</ul><h3><a id="HBase"></a>HBase</h3><p>The HBase REST API is provided by the Stargate service for HBase. See the HBase Stargate Setup section below for getting started with stargate and Knox with the Hortonworks Sandbox environment.</p><h4><a id="HBase+URL+Mapping"></a>HBase URL Mapping</h4>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase</code> </td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{stargate-host}:60080/</code> </td>
+    </tr>
+  </tbody>
+</table><h4><a id="HBase+Examples"></a>HBase Examples</h4><p>The examples below illustrate the set of basic operations with HBase instance using Stargate REST API. Use following link to get more more details about HBase/Stargate API: <a href="http://wiki.apache.org/hadoop/Hbase/Stargate">http://wiki.apache.org/hadoop/Hbase/Stargate</a>.</p><p>Note: Some HBase examples may not work due to enabled <a href="https://hbase.apache.org/book/hbase.accesscontrol.configuration.html">Access Control</a>. User may not be granted for performing operations in samples. In order to check if Access Control is configured in the HBase instance verify hbase-site.xml for a presence of <code>org.apache.hadoop.hbase.security.access.AccessController</code> in <code>hbase.coprocessor.master.classes</code> and <code>hbase.coprocessor.region.classes</code> properties.<br/>To grant the Read, Write, Create permissions to <code>guest</code> user execute the following command:</p>
 <pre><code>echo grant &#39;guest&#39;, &#39;RWC&#39; | hbase shell
-</code></pre><p>If you are using a cluster secured with Kerberos you will need to have used <code>kinit</code> to authenticate to the KDC </p><h3><a id="HBase+Stargate+Setup"></a>HBase Stargate Setup</h3><h4><a id="Launch+Stargate"></a>Launch Stargate</h4><p>The command below launches the Stargate daemon on port 60080</p>
+</code></pre><p>If you are using a cluster secured with Kerberos you will need to have used <code>kinit</code> to authenticate to the KDC </p><h4><a id="HBase+Stargate+Setup"></a>HBase Stargate Setup</h4><h4><a id="Launch+Stargate"></a>Launch Stargate</h4><p>The command below launches the Stargate daemon on port 60080</p>
 <pre><code>sudo /usr/lib/hbase/bin/hbase-daemon.sh start rest -p 60080
 </code></pre><p>Port 60080 is used because it was specified in sample Hadoop cluster deployment <code>{GATEWAY_HOME}/deployments/sandbox.xml</code>.</p><h4><a id="Configure+Sandbox+port+mapping+for+VirtualBox"></a>Configure Sandbox port mapping for VirtualBox</h4>
 <ol>
@@ -1957,11 +1955,13 @@ curl -i -k -u guest:guest-password -X DE
 <pre><code>sudo /usr/lib/hbase/bin/hbase-daemon.sh stop rest
 sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop regionserver
 sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop master
+sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop zookeeper
 
 sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start regionserver
 sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start master
+sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start zookeeper
 sudo /usr/lib/hbase/bin/hbase-daemon.sh start rest -p 60080
-</code></pre><h3><a id="HBase/Stargate+client+DSL"></a>HBase/Stargate client DSL</h3><p>For more details about client DSL usage please follow this [page|https://cwiki.apache.org/confluence/display/KNOX/Client+Usage].</p><h4><a id="systemVersion()+-+Query+Software+Version."></a>systemVersion() - Query Software Version.</h4>
+</code></pre><h4><a id="HBase/Stargate+client+DSL"></a>HBase/Stargate client DSL</h4><p>For more details about client DSL usage please follow this [page|https://cwiki.apache.org/confluence/display/KNOX/Client+Usage].</p><h4><a id="systemVersion()+-+Query+Software+Version."></a>systemVersion() - Query Software Version.</h4>
 <ul>
   <li>Request
   <ul>
@@ -2538,7 +2538,7 @@ session.shutdown(10, SECONDS)
   <tbody>
     <tr>
       <td>Gateway </td>
-      <td><code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code> </td>
+      <td><code>jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code> </td>
     </tr>
     <tr>
       <td>Cluster </td>
@@ -2552,23 +2552,20 @@ session.shutdown(10, SECONDS)
   <li>Make sure Hive Server is running in HTTP mode.</li>
   <li>Client side (JDBC):
   <ol>
-    <li>Hive JDBC in HTTP mode depends on following libraries to run successfully(must be in the classpath):
+    <li>Hive JDBC in HTTP mode depends on following minimal libraries set to run successfully(must be in the classpath):
     <ul>
-      <li>hadoop-common-2.2.0.2.0.6.0-76.jar;</li>
-      <li>hive-jdbc-0.12.0.2.0.6.0-76.jar;</li>
-      <li>hive-service-0.12.0.2.0.6.0-76.jar;</li>
+      <li>hive-jdbc-0.13.0.jar;</li>
+      <li>hive-service-0.13.0.jar;</li>
       <li>libthrift-0.9.0.jar;</li>
-      <li>httpcore-4.1.4.jar;</li>
-      <li>httpclient-4.1.3.jar;</li>
-      <li>hive-common-0.12.0.2.0.6.0-76.jar;</li>
-      <li>commons-logging-1.1.1.jar;</li>
+      <li>httpcore-4.2.5.jar;</li>
+      <li>httpclient-4.2.5.jar;</li>
+      <li>commons-logging-1.1.3.jar;</li>
+      <li>commons-codec-1.4.jar;</li>
       <li>slf4j-api-1.7.5.jar;</li>
       <li>slf4j-log4j12-1.7.5.jar;</li>
       <li>log4j-1.2.17.jar;</li>
-      <li>commons-codec-1.7.jar;</li>
     </ul></li>
-    <li>Import gateway certificate into the default JRE truststore.  It is located in the <code>/lib/security/cacerts</code>.  <code>keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore &lt;java-home&gt;/lib/security/cacerts</code>  Alternatively you can run your sample with additional parameters:  <code>-Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStore=&lt;path-to-trust-store&gt; -Djavax.net.ssl.trustStorePassword=&lt;trust-store-password&gt;</code></li>
-    <li>Connection URL has to be following:  <code>jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
+    <li>Connection URL has to be following: <code>jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive</code></li>
     <li>Look at <a href="https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations">https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations</a> for examples.  Hint: For testing it would be better to execute <code>set hive.security.authorization.enabled=false</code> as the first statement.  Hint: Good examples of Hive DDL/DML can be found here <a href="http://gettingstarted.hadooponazure.com/hw/hive.html">http://gettingstarted.hadooponazure.com/hw/hive.html</a></li>
   </ol></li>
 </ol><h5><a id="Customization"></a>Customization</h5><p>This example may need to be tailored to the execution environment. In particular host name, host port, user name, user password and context path may need to be changed to match your environment. In particular there is one example file in the distribution that may need to be customized. Take a moment to review this file. All of the values that may need to be customized can be found together at the top of the file.</p>
@@ -2596,8 +2593,10 @@ public class HiveJDBCSample {
       String password = user + &quot;-password&quot;;
       String gatewayHost = &quot;localhost&quot;;
       int gatewayPort = 8443;
+      String trustStore = &quot;/usr/lib/knox/data/security/keystores/gateway.jks&quot;;
+      String trustStorePassword = &quot;knoxsecret&quot;;
       String contextPath = &quot;gateway/sandbox/hive&quot;;
-      String connectionString = String.format( &quot;jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s&quot;, gatewayHost, gatewayPort, contextPath );
+      String connectionString = String.format( &quot;jdbc:hive2://%s:%d/;ssl=true;sslTrustStore=%s;trustStorePassword=%s?hive.server2.transport.mode=http;hive.server2.thrift.http.path=/%s&quot;, gatewayHost, gatewayPort, trustStore, trustStorePassword, contextPath );
 
       // load Hive JDBC Driver
       Class.forName( &quot;org.apache.hive.jdbc.HiveDriver&quot; );
@@ -2653,18 +2652,16 @@ public class HiveJDBCSample {
 }
 </code></pre><h6><a id="Groovy"></a>Groovy</h6><p>Make sure that GATEWAY_HOME/ext directory contains following libraries for successful execution:</p>
 <ul>
-  <li>hadoop-common-2.2.0.2.0.6.0-76.jar;</li>
-  <li>hive-jdbc-0.12.0.2.0.6.0-76.jar;</li>
-  <li>hive-service-0.12.0.2.0.6.0-76.jar;</li>
+  <li>hive-jdbc-0.13.0.jar;</li>
+  <li>hive-service-0.13.0.jar;</li>
   <li>libthrift-0.9.0.jar;</li>
-  <li>httpcore-4.1.4.jar;</li>
-  <li>httpclient-4.1.3.jar;</li>
-  <li>hive-common-0.12.0.2.0.6.0-76.jar;</li>
-  <li>commons-logging-1.1.1.jar;</li>
+  <li>httpcore-4.2.5.jar;</li>
+  <li>httpclient-4.2.5.jar;</li>
+  <li>commons-logging-1.1.3.jar;</li>
+  <li>commons-codec-1.4.jar;</li>
   <li>slf4j-api-1.7.5.jar;</li>
   <li>slf4j-log4j12-1.7.5.jar;</li>
   <li>log4j-1.2.17.jar;</li>
-  <li>commons-codec-1.7.jar;</li>
 </ul><p>There are several ways to execute this sample depending upon your preference.</p><p>You can use the Groovy interpreter provided with the distribution.</p>
 <pre><code>java -jar bin/shell.jar samples/hive/groovy/jdbc/sandbox/HiveJDBCSample.groovy
 </code></pre><p>You can manually type in the KnoxShell DSL script into the interactive Groovy interpreter provided with the distribution.</p>
@@ -2676,8 +2673,10 @@ user = &quot;guest&quot;;
 password = user + &quot;-password&quot;;
 gatewayHost = &quot;localhost&quot;;
 gatewayPort = 8443;
+trustStore = &quot;/usr/lib/knox/data/security/keystores/gateway.jks&quot;;
+trustStorePassword = &quot;knoxsecret&quot;;
 contextPath = &quot;gateway/sandbox/hive&quot;;
-connectionString = String.format( &quot;jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s&quot;, gatewayHost, gatewayPort, contextPath );
+connectionString = String.format( &quot;jdbc:hive2://%s:%d/;ssl=true;sslTrustStore=%s;trustStorePassword=%s?hive.server2.transport.mode=http;hive.server2.thrift.http.path=/%s&quot;, gatewayHost, gatewayPort, trustStore, trustStorePassword, contextPath );
 
 // Load Hive JDBC Driver
 Class.forName( &quot;org.apache.hive.jdbc.HiveDriver&quot; );
@@ -2737,7 +2736,7 @@ connection.close();
 2012-02-03 --- 18:35:34 --- SampleClass6 --- [TRACE]
 2012-02-03 --- 18:35:34 --- SampleClass2 --- [DEBUG]
 ...
-</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT Request Payload Size Restriction</h3><p>With one exception there are no know size limits for requests or responses payloads that pass through the gateway. The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego. There is an undocumented configuration setting to modify this limit&rsquo;s value if required. In the future this will be made more easily configuration and at that time it will be documented.</p><h3><a id="LDAP+Groups+Acquisition"></a>LDAP Groups Acquisition</h3><p>The LDAP authenticator currently does not &ldquo;out of the box&rdquo; support the acquisition of 
 group information. This can be addressed by implementing a custom Shiro Realm extension. Building this into the default implementation is on the roadmap.</p><h3><a id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups that are acquired via Identity Assertion Group Principal Mapping are not propigated to the Hadoop services. Therefore groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding Logs</h3><p>When things aren&rsquo;t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output is written directly to the console. If you want to ca
 pture that output you will need to redirect the console output to a file using OS specific techniques.</p>
+</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT Request Payload Size Restriction</h3><p>With one exception there are no know size limits for requests or responses payloads that pass through the gateway. The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this one case there is currently a 4Kb payload size limit for the first request made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship between itself and the cluster via SPNego. There is an undocumented configuration setting to modify this limit&rsquo;s value if required. In the future this will be made more easily configuration and at that time it will be documented.</p><h3><a id="LDAP+Groups+Acquisition+from+AD"></a>LDAP Groups Acquisition from AD</h3><p>The LDAP authenticator currently does not &ldquo;out of the box&rdquo; support the
  acquisition of group information from Microsoft Active Directory. Building this into the default implementation is on the roadmap.</p><h3><a id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups that are acquired via Shiro Group Lookup and/or Identity Assertion Group Principal Mapping are not propagated to the Hadoop services. Therefore groups used for Service Level Authorization policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding Logs</h3><p>When things aren&rsquo;t working the first thing you need to do is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java -jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output is written directly to the console. If you want t
 o capture that output you will need to redirect the console output to a file using OS specific techniques.</p>
 <pre><code>java -jar bin/gateway.jar &gt; gateway.log
 </code></pre><h4><a id="bin/gateway.sh+start"></a>bin/gateway.sh start</h4><p>When the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out and /var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a id="Increasing+Logging"></a>Increasing Logging</h3><p>The <code>log4j.properties</code> files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of the logging done by Knox. The Knox server must be restarted in order for these changes to take effect. There are various useful loggers pre-populated but commented out.</p>
 <pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging of Apache Knox itself.
@@ -2848,7 +2847,7 @@ If this HTTPS server uses a certificate 
  not match the domain name in the URL).
 If you&#39;d like to turn off curl&#39;s verification of the certificate, use
  the -k (or --insecure) option.
-</code></pre><h3><a id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs can be filed using <a href="https://issues.apache.org/jira/browse/KNOX">Jira</a>. Please include the results of this command below in the Environment section. Also include the version of Hadoop being used in the same section.</p>
+</code></pre><h3><a id="SPNego+Authentication+Issues"></a>SPNego Authentication Issues</h3><p>Calls from Knox to Secure Hadoop Cluster fails, with SPNego authentication problems, if there was a TGT for knox in disk cache when Knox was started.</p><p>You are likely to run into this situation on developer machines where develeoper could have knited for some testing.</p><p>Work Around: clear TGT of Knox from disk cache ( calling kdestroy would do it), before starting knox</p><h3><a id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs can be filed using <a href="https://issues.apache.org/jira/browse/KNOX">Jira</a>. Please include the results of this command below in the Environment section. Also include the version of Hadoop being used in the same section.</p>
 <pre><code>cd {GATEWAY_HOME}
 java -jar bin/gateway.jar -version
 </code></pre><h2><a id="Export+Controls"></a>Export Controls</h2><p>Apache Knox Gateway includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country&rsquo;s laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <a href="http://www.wassenaar.org">http://www.wassenaar.org</a> for more information.</p><p>The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC
  Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.</p><p>The following provides more details on the included cryptographic software:</p>

Modified: knox/site/books/knox-0-5-0/likeised
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-5-0/likeised?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/site/books/knox-0-5-0/likeised (original)
+++ knox/site/books/knox-0-5-0/likeised Thu May 22 14:30:00 2014
@@ -41,4 +41,4 @@ s@<h2><a id="Export+Controls@</div><div 
 # closing the last chapter section, page-wrap and content sections is done outside of this script
 # using cat >> filename
 
-# sed -f likeised knox-incubating-0-4-0.html > knox-incubating-0-4-0-new.html && echo "</div></div></div>" >> knox-incubating-0-4-0-new.html
+# sed -f likeised knox-0-5-0.html > knox-0-5-0-new.html && echo "</div></div></div>" >> knox-0-5-0-new.html



Mime
View raw message