knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lmc...@apache.org
Subject svn commit: r1596890 [2/2] - in /knox: site/books/knox-0-5-0/ trunk/books/0.5.0/
Date Thu, 22 May 2014 14:30:01 GMT
Modified: knox/trunk/books/0.5.0/book.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book.md (original)
+++ knox/trunk/books/0.5.0/book.md Thu May 22 14:30:00 2014
@@ -30,7 +30,6 @@
 * #[Apache Knox Details]
     * #[Apache Knox Directory Layout]
     * #[Supported Services]
-    * #[Configure Sandbox port mapping for VirtualBox]
 * #[Gateway Details]
     * #[Configuration]
     * #[Knox CLI]

Modified: knox/trunk/books/0.5.0/book_client-details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_client-details.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_client-details.md (original)
+++ knox/trunk/books/0.5.0/book_client-details.md Thu May 22 14:30:00 2014
@@ -81,7 +81,7 @@ Using `^C` to exit can sometimes leaves 
 
 The shell can also be used to execute a script by passing a single filename argument.
 
-    java -jar bin/shell.jar samples/ExampleWebHdfsPutGetFile.groovy
+    java -jar bin/shell.jar samples/ExampleWebHdfsPutGet.groovy
 
 
 ### Examples ###
@@ -487,27 +487,27 @@ The JARs currently required by the clien
 
 So on Linux/MacOS you would need this command
 
-    groovy -cp lib/gateway-shell-0.2.0-SNAPSHOT.jar:dep/httpclient-4.2.3.jar:dep/httpcore-4.2.2.jar:dep/commons-lang3-3.1.jar:dep/commons-codec-1.7.jar
samples/ExampleWebHdfsPutGet.groovy
+    groovy -cp lib/gateway-shell-0.4.0.jar:dep/httpclient-4.2.5.jar:dep/httpcore-4.2.4.jar:dep/commons-lang3-3.1.jar:dep/commons-codec-1.7.jar
samples/ExampleWebHdfsPutGet.groovy
 
 and on Windows you would need this command
 
-    groovy -cp lib/gateway-shell-0.2.0-SNAPSHOT.jar;dep/httpclient-4.2.3.jar;dep/httpcore-4.2.2.jar;dep/commons-lang3-3.1.jar;dep/commons-codec-1.7.jar
samples/ExampleWebHdfsPutGet.groovy
+    groovy -cp lib/gateway-shell-0.4.0.jar;dep/httpclient-4.2.5.jar;dep/httpcore-4.2.4.jar;dep/commons-lang3-3.1.jar;dep/commons-codec-1.7.jar
samples/ExampleWebHdfsPutGet.groovy
 
 The exact list of required JARs is likely to change from release to release so it is recommended
that you utilize the wrapper `bin/shell.jar`.
 
 In addition because the DSL can be used via standard Groovy, the Groovy integrations in many
popular IDEs (e.g. IntelliJ , Eclipse) can also be used.
 This makes it particularly nice to develop and execute scripts to interact with Hadoop.
 The code-completion features in modern IDEs in particular provides immense value.
-All that is required is to add the shell-0.2.0.jar to the projects class path.
+All that is required is to add the gateway-shell-0.4.0.jar to the projects class path.
 
 There are a variety of Groovy tools that make it very easy to work with the standard interchange
formats (i.e. JSON and XML).
 In Groovy the creation of XML or JSON is typically done via a "builder" and parsing done
via a "slurper".
 In addition once JSON or XML is "slurped" the GPath, an XPath like feature build into Groovy
can be used to access data.
 
 * XML
-    * Markup Builder [Overview]http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder),
[API](http://groovy.codehaus.org/api/groovy/xml/MarkupBuilder.html)
-    * XML Slurper [Overview]http://groovy.codehaus.org/Reading+XML+using+Groovy's+XmlSlurper),
[API](http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html)
-    * XPath [Overview]http://groovy.codehaus.org/GPath), [API](http://docs.oracle.com/javase/1.5.0/docs/api/javax/xml/xpath/XPath.html)
+    * Markup Builder [Overview](http://groovy.codehaus.org/Creating+XML+using+Groovy's+MarkupBuilder),
[API](http://groovy.codehaus.org/api/groovy/xml/MarkupBuilder.html)
+    * XML Slurper [Overview](http://groovy.codehaus.org/Reading+XML+using+Groovy's+XmlSlurper),
[API](http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html)
+    * XPath [Overview](http://groovy.codehaus.org/GPath), [API](http://docs.oracle.com/javase/1.5.0/docs/api/javax/xml/xpath/XPath.html)
 * JSON
     * JSON Builder [API](http://groovy.codehaus.org/gapi/groovy/json/JsonBuilder.html)
     * JSON Slurper [API](http://groovy.codehaus.org/gapi/groovy/json/JsonSlurper.html)

Modified: knox/trunk/books/0.5.0/book_gateway-details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_gateway-details.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_gateway-details.md (original)
+++ knox/trunk/books/0.5.0/book_gateway-details.md Thu May 22 14:30:00 2014
@@ -46,7 +46,7 @@ These default topology URLs exist for al
 
 #### Fully Qualified URLs #####
 Examples of mappings for the WebHDFS, WebHCat, Oozie and Stargate/HBase are shown below.
-These mapping are generated from the combination of the gateway configuration file (i.e.
`{GATEWAY_HOME}/conf/gateway-site.xml`) and the cluster topology descriptors (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
+These mapping are generated from the combination of the gateway configuration file (i.e.
`{GATEWAY_HOME}/conf/gateway-site.xml`) and the cluster topology descriptors (e.g. `{GATEWAY_HOME}/conf/topologies/{cluster-name}.xml`).
 The port numbers show for the Cluster URLs represent the default ports for these services.
 The actual port number may be different for a given cluster.
 
@@ -63,7 +63,7 @@ The actual port number may be different 
     * Gateway: `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase`
     * Cluster: `http://{hbase-host}:60080`
 * Hive JDBC
-    * Gateway: `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
+    * Gateway: `jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
     * Cluster: `http://{hive-host}:10001/cliservice`
 
 The values for `{gateway-host}`, `{gateway-port}`, `{gateway-path}` are provided via the
gateway configuration file (i.e. `{GATEWAY_HOME}/conf/gateway-site.xml`).
@@ -72,7 +72,7 @@ The value for `{cluster-name}` is derive
 
 The value for `{webhdfs-host}`, `{webhcat-host}`, `{oozie-host}`, `{hbase-host}` and `{hive-host}`
are provided via the cluster topology descriptor (e.g. `{GATEWAY_HOME}/deployments/{cluster-name}.xml`).
 
-Note: The ports 50070, 50111, 11000, 60080 and 10001 are the defaults for WebHDFS, WebHCat,
Oozie, Stargate/HBase and Hive respectively.
+Note: The ports 50070, 50111, 11000, 60080 (default 8080) and 10001 are the defaults for
WebHDFS, WebHCat, Oozie, Stargate/HBase and Hive respectively.
 Their values can also be provided via the cluster topology descriptor if your Hadoop cluster
uses different ports.
 
 <<config.md>>
@@ -86,4 +86,3 @@ Their values can also be provided via th
 <<config_webappsec_provider.md>>
 <<config_preauth_sso_provider.md>>
 <<config_audit.md>>
-

Modified: knox/trunk/books/0.5.0/book_getting-started.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_getting-started.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_getting-started.md (original)
+++ knox/trunk/books/0.5.0/book_getting-started.md Thu May 22 14:30:00 2014
@@ -21,10 +21,10 @@ This section provides everything you nee
 
 #### Hadoop ####
 
-An an existing Hadoop 1.x or 2.x cluster is required for Knox sit in front of and protect.
+An existing Hadoop 2.x cluster is required for Knox 0.4.0 to sit in front of and protect.
 It is possible to use a Hadoop cluster deployed on EC2 but this will require additional configuration
not covered here.
-It is also possible to use a limited set of services in Hadoop cluster secured with Kerberos.
-This too required additional configuration that is not described here.
+It is also possible to protect access to a services of a Hadoop cluster that is secured with
kerberos.
+This too requires additional configuration that is described in other sections of this guide.
 See #[Supported Services] for details on what is supported for this release.
 
 The Hadoop cluster should be ensured to have at least WebHDFS, WebHCat (i.e. Templeton) and
Oozie configured, deployed and running.
@@ -40,57 +40,46 @@ All of the instructions and samples prov
 
 #### Apache Knox Directory Layout ####
 
-Knox can be installed by expanding the zip file or with rpm. With rpm based install the following
directories are created in addition to those described in
-this section.
+Knox can be installed by expanding the zip/archive file.
 
-    /usr/lib/knox
-    /var/log/knox
-    /var/run/knox
-
-The directory `/usr/lib/knox` is considered your `{GATEWAY_HOME}` and will adhere to the
layout described below.
-The directory `/var/log/knox` will contain the output files from the server.
-The directory `/var/run/knox` will contain the process ID for a currently running gateway
server.
-
-
-Regardless of the installation method used the layout and content of the `{GATEWAY_HOME}`
will be identical.
 The table below provides a brief explanation of the important files and directories within
`{GATEWWAY_HOME}`
 
 | Directory     | Purpose |
 | ------------- | ------- |
 | conf/         | Contains configuration files that apply to the gateway globally (i.e. not
cluster specific ).       |
-| bin/          | Contains the executable shell scripts, batch files and JARs for clients
and servers.                |
-| deployments/  | Contains topology descriptors used to configure the gateway for specific
Hadoop clusters.           |
-| lib/          | Contains the JARs for all the components that make up the gateway.    
                             |
-| dep/          | Contains the JARs for all of the components upon which the gateway depends.
                        |
-| ext/          | A directory where user supplied extension JARs can be placed to extends
the gateways functionality. |
-| samples/      | Contains a number of samples that can be used to explore the functionality
of the gateway.          |
-| templates/    | Contains default configuration files that can be copied and customized.
                            |
-| README        | Provides basic information about the Apache Knox Gateway.             
                             |
-| ISSUES        | Describes significant know issues.                                    
                             |
-| CHANGES       | Enumerates the changes between releases.                              
                             |
-| LICENSE       | Documents the license under which this software is provided.          
                             |
-| NOTICE        | Documents required attribution notices for included dependencies.     
                             |
-| DISCLAIMER    | Documents that this release is from a project undergoing incubation at
Apache.                      |
+| data/         | Contains security and topology specific artifacts that require read/write
access at runtime |
+|data/topologies/|Contains topology files that represent Hadoop clusters which the gateway
uses to deploy cluster proxies|
+|data/security/ | Contains the persisted master secret and keystore dir|
+|data/security/keystores/| Contains the gateway identity keystore and credential stores for
the gateway and each deployed cluster topology|
+| bin/          | Contains the executable shell scripts, batch files and JARs for clients
and servers.|
+| data/deployments/ | Contains deployed cluster topologies used to protect access to specific
Hadoop clusters.|
+| lib/          | Contains the JARs for all the components that make up the gateway.|
+| dep/          | Contains the JARs for all of the components upon which the gateway depends.|
+| ext/          | A directory where user supplied extension JARs can be placed to extends
the gateways functionality.|
+| pids/         | Contains the process ids for running ldap and gateway servers|
+| samples/      | Contains a number of samples that can be used to explore the functionality
of the gateway.|
+| templates/    | Contains default configuration files that can be copied and customized.|
+| README        | Provides basic information about the Apache Knox Gateway.|
+| ISSUES        | Describes significant know issues.|
+| CHANGES       | Enumerates the changes between releases.|
+| LICENSE       | Documents the license under which this software is provided.|
+| NOTICE        | Documents required attribution notices for included dependencies.|
+| DISCLAIMER    | Documents that this release is from a project undergoing incubation at
Apache.|
 
 
 ### Supported Services ###
 
 This table enumerates the versions of various Hadoop services that have been tested to work
with the Knox Gateway.
-Only more recent versions of some Hadoop components when secured via Kerberos can be accessed
via the Knox Gateway.
 
 | Service            | Version    | Non-Secure  | Secure |
 | ------------------ | ---------- | ----------- | ------ |
-| WebHDFS            | 2.1.0      | ![y]        | ![y]   |
-| WebHCat/Templeton  | 0.11.0     | ![y]        | ![n]   |
+| WebHDFS            | 2.4.0      | ![y]        | ![y]   |
+| WebHCat/Templeton  | 0.13.0     | ![y]        | ![y]   |
 |                    | 0.12.0     | ![y]        | ![y]   |
 | Ozzie              | 4.0.0      | ![y]        | ![y]   |
-| HBase/Stargate     | 0.95.2     | ![y]        | ![n]   |
-| Hive (via WebHCat) | 0.11.0     | ![y]        | ![n]   |
-|                    | 0.12.0     | ![y]        | ![y]   |
-| Hive (via JDBC)    | 0.11.0     | ![n]        | ![n]   |
-|                    | 0.12.0     | ![y]        | ![n]   |
-| Hive (via ODBC)    | 0.11.0     | ![n]        | ![n]   |
-|                    | 0.12.0     | ![n]        | ![n]   |
+| HBase/Stargate     | 0.98.0     | ![y]        | ![y]   |
+| Hive (via WebHCat) | 0.13.0     | ![y]        | ![y]   |
+| Hive (via JDBC)    | 0.13.0     | ![y]        | ![y]   |
 
 
 ### More Examples ###

Modified: knox/trunk/books/0.5.0/book_limitations.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_limitations.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_limitations.md (original)
+++ knox/trunk/books/0.5.0/book_limitations.md Thu May 22 14:30:00 2014
@@ -28,15 +28,14 @@ There is an undocumented configuration s
 In the future this will be made more easily configuration and at that time it will be documented.
 
 
-### LDAP Groups Acquisition ###
+### LDAP Groups Acquisition from AD ###
 
-The LDAP authenticator currently does not "out of the box" support the acquisition of group
information.
-This can be addressed by implementing a custom Shiro Realm extension.
+The LDAP authenticator currently does not "out of the box" support the acquisition of group
information from Microsoft Active Directory.
 Building this into the default implementation is on the roadmap.
 
 
 ### Group Membership Propagation ###
 
-Groups that are acquired via Identity Assertion Group Principal Mapping are not propigated
to the Hadoop services.
+Groups that are acquired via Shiro Group Lookup and/or Identity Assertion Group Principal
Mapping are not propagated to the Hadoop services.
 Therefore groups used for Service Level Authorization policy may not match those acquired
within the cluster via GroupMappingServiceProvider plugins.
 

Modified: knox/trunk/books/0.5.0/book_troubleshooting.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/book_troubleshooting.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/book_troubleshooting.md (original)
+++ knox/trunk/books/0.5.0/book_troubleshooting.md Thu May 22 14:30:00 2014
@@ -293,6 +293,15 @@ Curl will present you with the follow me
 	 the -k (or --insecure) option.
 
 
+### SPNego Authentication Issues ###
+
+Calls from Knox to Secure Hadoop Cluster fails, with SPNego authentication problems,
+if there was a TGT for knox in disk cache when Knox was started.
+
+You are likely to run into this situation on developer machines where develeoper could have
knited for some testing.
+
+Work Around: clear TGT of Knox from disk cache ( calling kdestroy would do it), before starting
knox
+
 ### Filing Bugs ###
 
 Bugs can be filed using [Jira][jira].

Modified: knox/trunk/books/0.5.0/config.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/config.md (original)
+++ knox/trunk/books/0.5.0/config.md Thu May 22 14:30:00 2014
@@ -114,9 +114,9 @@ topology/service/url
 
 #### Hostmap Provider ####
 
-The purpose of the Hostmap provider is to handle situations where host are know by one name
within the cluster and another name externally.
-This frequently occurs when virtual machines are used and in particular using cloud hosting
services.
-Currently the Hostmap provider is configured as part of the topology file.
+The purpose of the Hostmap provider is to handle situations where host are known by one name
within the cluster and another name externally.
+This frequently occurs when virtual machines are used and in particular when using cloud
hosting services.
+Currently, the Hostmap provider is configured as part of the topology file.
 The basic structure is shown below.
 
     <topology>
@@ -176,7 +176,7 @@ The Hostmap configuration required to al
 
 ##### Hostmap Provider Example - Sandbox #####
 
-Hortonwork's Sandbox 2.x poses a different challenge for host name mapping.
+The Hortonworks Sandbox 2.x poses a different challenge for host name mapping.
 This version of the Sandbox uses port mapping to make the Sandbox VM appear as though it
is accessible via localhost.
 However the Sandbox VM is internally configured to consider sandbox.hortonworks.com as the
host name.
 So from the perspective of a client accessing Sandbox the external host name is localhost.
@@ -264,21 +264,21 @@ The following is a description of how th
 
 Upon start of the gateway server we:
 
-1. Look for an identity store at `conf/security/keystores/gateway.jks`.
+1. Look for an identity store at `data/security/keystores/gateway.jks`.
    The identity store contains the certificate and private key used to represent the identity
of the server for SSL connections and signature creation.
     * If there is no identity store we create one and generate a self-signed certificate
for use in standalone/demo mode.
       The certificate is stored with an alias of gateway-identity.
-    * If there is an identity store found than we ensure that it can be loaded using the
provided master secret and that there is an alias with called gateway-identity.
-2. Look for a credential store at `conf/security/keystores/__gateway-credentials.jceks`.
+    * If there is an identity store found than we ensure that it can be loaded using the
provided master secret and that there is an alias called gateway-identity.
+2. Look for a credential store at `data/security/keystores/__gateway-credentials.jceks`.
    This credential store is used to store secrets/passwords that are used by the gateway.
-   For instance, this is where the pass-phrase for accessing the gateway-identity certificate
is kept.
-    * If there is no credential store found then we create one and populate it with a generated
pass-phrase for the alias `gateway-identity-passphrase`.
+   For instance, this is where the passphrase for accessing the gateway-identity certificate
is kept.
+    * If there is no credential store found then we create one and populate it with a generated
passphrase for the alias `gateway-identity-passphrase`.
       This is coordinated with the population of the self-signed cert into the identity-store.
     * If a credential store is found then we ensure that it can be loaded using the provided
master secret and that the expected aliases have been populated with secrets.
 
 Upon deployment of a Hadoop cluster topology within the gateway we:
 
-1. Look for a credential store for the topology. For instance, we have a sample topology
that gets deployed out of the box.  We look for `conf/security/keystores/sandbox-credentials.jceks`.
This topology specific credential store is used for storing secrets/passwords that are used
for encrypting sensitive data with topology specific keys.
+1. Look for a credential store for the topology. For instance, we have a sample topology
that gets deployed out of the box.  We look for `data/security/keystores/sandbox-credentials.jceks`.
This topology specific credential store is used for storing secrets/passwords that are used
for encrypting sensitive data with topology specific keys.
     * If no credential store is found for the topology being deployed then one is created
for it.
       Population of the aliases is delegated to the configured providers within the system
that will require the use of a  secret for a particular task.
       They may programmatic set the value of the secret or choose to have the value for the
specified alias generated through the AliasService.
@@ -296,7 +296,6 @@ See the Knox CLI section for description
 In order to provide your own certificate for use by the gateway, you will need to either
import an existing key pair into a Java keystore or generate a self-signed cert using the
Java keytool.
 
 ##### Importing a key pair into a Java keystore #####
-# ----NEEDS TESTING
 One way to accomplish this is to start with a PKCS12 store for your key pair and then convert
it to a Java keystore or JKS.
 
     openssl pkcs12 -export -in cert.pem -inkey key.pem > server.p12
@@ -311,11 +310,9 @@ While using this approach a couple of im
 
 1. the alias MUST be "gateway-identity"
 2. the name of the expected identity keystore for the gateway MUST be gateway.jks
-3. the passwords for the keystore and the imported key MUST both be the master secret for
the gateway install
+3. the passwords for the keystore and the imported key may both be set to the master secret
for the gateway install
 
-NOTE: The password for the keystore as well as that of the imported key must be the master
secret for the gateway instance.
-
-# ----END NEEDS TESTING
+NOTE: The password for the keystore as well as that of the imported key may be the master
secret for the gateway instance or you may set the gateway-identity-passphrase alias using
the Knox CLI to the actual key passphrase. See the Knox CLI section for details.
 
 ##### Generating a self-signed cert for use in testing or development environments #####
 
@@ -326,21 +323,21 @@ Keytool will prompt you for a number of 
 
 *NOTE:* When it prompts you for your First and Last name be sure to type in the hostname
of the machine that your gateway instance will be running on. This is used by clients during
hostname verification to ensure that the presented certificate matches the hostname that was
used in the URL for the connection - so they need to match.
 
-*NOTE:* When it prompts for the key password just press enter to ensure that it is the same
as the keystore password. Which as was described earlier must match the master secret for
the gateway instance.
+*NOTE:* When it prompts for the key password just press enter to ensure that it is the same
as the keystore password. Which as was described earlier must match the master secret for
the gateway instance. Alternatively, you can set it to another passphrase - take note of it
and set the gateway-identity-passphrase alias to that passphrase using the Knox CLI.
 
 See the Knox CLI section for descriptions of the command line utilties related to the management
of the keystores.
 
 ##### Credential Store #####
-Whenever you provide your own keystore with either a self-signed cert or a real certificate
signed by a trusted authority, you will need to create an empty credential store. This is
necessary for the current release in order for the system to utilize the same password for
the keystore and the key.
+Whenever you provide your own keystore with either a self-signed cert or an issued certificate
signed by a trusted authority, you will need to set an alias for the gateway-identity-passphrase
or create an empty credential store. This is necessary for the current release in order for
the system to determine the correct password for the keystore and the key.
 
 The credential stores in Knox use the JCEKS keystore type as it allows for the storage of
general secrets in addition to certificates.
 
-Keytool may be used to create credential stores but the Knox CLI section details how to create
aliases. These aliases are managed within credential stores which are created by the CLI as
appropriate. 
+Keytool may be used to create credential stores but the Knox CLI section details how to create
aliases. These aliases are managed within credential stores which are created by the CLI as
needed. The simplest approach is to create the gateway-identity-passpharse alias with the
Knox CLI. This will create the credential store if it doesn't already exist and add the key
passphrase.
 
 See the Knox CLI section for descriptions of the command line utilties related to the management
of the credential stores.
 
 ##### Provisioning of Keystores #####
-Once you have created these keystores you must move them into place for the gateway to discover
them and use them to represent its identity for SSL connections. This is done by copying the
keystores to the `{GATEWAY_HOME}/conf/security/keystores` directory for your gateway install.
+Once you have created these keystores you must move them into place for the gateway to discover
them and use them to represent its identity for SSL connections. This is done by copying the
keystores to the `{GATEWAY_HOME}/data/security/keystores` directory for your gateway install.
 
 #### Summary of Secrets to be Managed ####
 

Modified: knox/trunk/books/0.5.0/config_kerberos.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_kerberos.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/config_kerberos.md (original)
+++ knox/trunk/books/0.5.0/config_kerberos.md Thu May 22 14:30:00 2014
@@ -58,7 +58,7 @@ You could use * for local developer test
         <value>FQDN_OF_KNOX_HOST</value>
     </property>
 
-#### Grant proxy privilege for Knox in `webhcat-stie.xml` on Hadoop master nodes ####
+#### Grant proxy privilege for Knox in `webhcat-site.xml` on Hadoop master nodes ####
 
 Update `webhcat-site.xml` and add the following lines towards the end of the file.
 
@@ -74,7 +74,7 @@ You could use * for local developer test
         <value>FQDN_OF_KNOX_HOST</value>
     </property>
 
-#### Grant proxy privilege for Knox in `oozie-stie.xml` on Oozie host ####
+#### Grant proxy privilege for Knox in `oozie-site.xml` on Oozie host ####
 
 Update `oozie-site.xml` and add the following lines towards the end of the file.
 
@@ -90,6 +90,36 @@ You could use * for local developer test
        <value>FQDN_OF_KNOX_HOST</value>
     </property>
 
+#### Enable http transport mode and use substitution in Hive Server2 ####
+
+Update `hive-site.xml` and set the following properties on Hive Server2 hosts.
+Some of the properties may already be in the hive-site.xml. 
+Ensure that the values match the ones below.
+
+<property>
+  <name>hive.server2.allow.user.substitution</name>
+  <value>true</value>
+</property>
+
+<property>
+	<name>hive.server2.transport.mode</name>
+	<value>http</value>
+	<description>Server transport mode. "binary" or "http".</description>
+</property>
+
+<property>
+	<name>hive.server2.thrift.http.port</name>
+	<value>10001</value>
+	<description>Port number when in HTTP mode.</description>
+</property>
+
+<property>
+	<name>hive.server2.thrift.http.path</name>
+	<value>cliservice</value>
+	<description>Path component of URL endpoint when in HTTP mode.</description>
+</property>
+
+
 #### Copy knox keytab to Knox host ####
 
 Add unix account for the knox user on Knox host

Modified: knox/trunk/books/0.5.0/config_preauth_sso_provider.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_preauth_sso_provider.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/config_preauth_sso_provider.md (original)
+++ knox/trunk/books/0.5.0/config_preauth_sso_provider.md Thu May 22 14:30:00 2014
@@ -25,6 +25,8 @@ Knox Gateway needs a pluggable mechanism
 
 #### Configuration ####
 ##### Overview #####
+This provider was designed for use with identity solutions such as those provided by CA's
SiteMinder and IBM's Tivoli Access Manager. While direct testing with these products has not
been done, there has been extensive unit and functional testing that ensure that it should
work with such providers.
+
 The HeaderPreAuth provider is configured within the topology file and has a minimal configuration
that assumes SM_USER for CA SiteMinder. The following example is the bare minimum configuration
for SiteMinder (with no IP address validation).
 
 	<provider>

Modified: knox/trunk/books/0.5.0/knox_cli.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/knox_cli.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/knox_cli.md (original)
+++ knox/trunk/books/0.5.0/knox_cli.md Thu May 22 14:30:00 2014
@@ -26,11 +26,15 @@ The knoxcli.sh script is located in the 
 ##### knoxcli.sh [--help] #####
 prints help for all commands
 
+#### Knox Verison Info ####
+##### knoxcli.sh version [--help] #####
+Displays Knox version information.
+
 #### Master secret persistence ####
-##### knoxcli.sh create-master [--help] #####
+##### knoxcli.sh create-master [--force][--help] #####
 Creates and persists an encrypted master secret in a file within {GATEWAY_HOME}/data/security/master.

 
-NOTE: This command fails when there is an existing master file in the expected location.
+NOTE: This command fails when there is an existing master file in the expected location.
You may force it to overwrite the master file with the --force switch. NOTE: this will require
you to change passwords protecting the keystores for the gateway identity keystores and all
credential stores.
 
 #### Alias creation ####
 ##### knoxcli.sh create-alias n [--cluster c] [--value v] [--generate] [--help] #####
@@ -68,3 +72,7 @@ argument | description
 :--------|-----------
 --hostname	|	name of the host to be used in the self-signed certificate. This allows multi-host
deployments to specify the proper hostnames for hostname verification to succeed on the client
side of the SSL connection. The default is “localhost”.
 
+#### Topology Redeploy ####
+#### redeploy [--cluster c] ####
+Redeploys one or all of the gateway's clusters (a.k.a topologies).
+

Modified: knox/trunk/books/0.5.0/quick_start.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/quick_start.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/quick_start.md (original)
+++ knox/trunk/books/0.5.0/quick_start.md Thu May 22 14:30:00 2014
@@ -41,34 +41,29 @@ Use the command below to check the versi
 
 #### Hadoop ####
 
-Knox supports Hadoop 1.x or 2.x, the quick start instructions assume a Hadoop 2.x virtual
machine based environment. 
+Knox 0.4.0 supports Hadoop 2.x, the quick start instructions assume a Hadoop 2.x virtual
machine based environment. 
 
 
 ### 2 - Download Hadoop 2.x VM ###
 The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine
[Sandbox](http://hortonworks.com/products/hdp-2/#install). Please note Knox supports other
Hadoop distributions and is configurable against a full blown Hadoop cluster.
-Configuring Knox for Hadoop 1.x/2.x version, or Hadoop deployed in EC2 or a custom Hadoop
cluster is documented in advance deployment guide.
+Configuring Knox for Hadoop 2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster
is documented in advance deployment guide.
 
 
 ### 3 - Download Apache Knox Gateway ###
 
 Download one of the distributions below from the [Apache mirrors][mirror].
 
-* Source archive: [knox-incubating-0.4.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1
digest][src-sha], [MD5 digest][src-md5])
-* Binary archive: [knox-incubating-0.4.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha],
[MD5 digest][bin-md5])
-* RPM package: [knox-incubating-0.4.0.rpm][rpm] ([PGP signature][rpm-pgp], [SHA1 digest][rpm-sha],
[MD5 digest][rpm-md5])
-
-[src-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip
-[src-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.sha
-[src-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating-src.zip.asc
-[src-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.md5
-[bin-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip
-[bin-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.asc
-[bin-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.sha
-[bin-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.md5
-[rpm]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm
-[rpm-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.sha
-[rpm-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating.rpm.asc
-[rpm-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.md5
+* Source archive: [knox-0.4.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha],
[MD5 digest][src-md5])
+* Binary archive: [knox-0.4.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha],
[MD5 digest][bin-md5])
+
+[src-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip
+[src-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.sha
+[src-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.asc
+[src-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0-src.zip.md5
+[bin-zip]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip
+[bin-pgp]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.asc
+[bin-sha]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.sha
+[bin-md5]: http://www.apache.org/dyn/closer.cgi/knox/0.4.0/knox-0.4.0.zip.md5
 
 Apache Knox Gateway releases are available under the [Apache License, Version 2.0][asl].
 See the NOTICE file contained in each release artifact for applicable copyright attribution
notices.
@@ -85,17 +80,17 @@ Make sure you get these files from the m
 Then verify the signatures using one of the methods below.
 
     % pgpk -a KEYS
-    % pgpv knox-incubating-0.4.0.zip.asc
+    % pgpv knox-0.4.0.zip.asc
 
 or
 
     % pgp -ka KEYS
-    % pgp knox-incubating-0.4.0.zip.asc
+    % pgp knox-0.4.0.zip.asc
 
 or
 
     % gpg --import KEYS
-    % gpg --verify knox-incubating-0.4.0.zip.asc
+    % gpg --verify knox-0.4.0.zip.asc
 
 ### 4 - Start Hadoop virtual machine ###
 
@@ -114,25 +109,10 @@ The example below provides a command tha
 Note the `{VERSION}` portion of the command must be replaced with an actual Apache Knox Gateway
version number.
 This might be 0.4.0 for example and must patch the value in the file downloaded.
 
-    jar xf knox-incubating-{VERSION}.zip
-
-This will create a directory `knox-incubating-{VERSION}` in your current directory.
-The directory `knox-incubating-{VERSION}` will considered your `{GATEWAY_HOME}`
-
-
-#### RPM ####
-
-If you downloaded the RPM distribution you can install it using normal RPM package tools.
-It is important that the user that will be running the gateway server is used to install.
-This is because several directories are created that are owned by this user.
-These command will install Knox to `/usr/lib/knox` following the pattern of other Hadoop
components.
-This directory will be considered your `{GATEWAY_HOME}`.
-
-    sudo yum localinstall knox-incubating-{VERSION}.rpm
-
-or
+    jar xf knox-{VERSION}.zip
 
-    sudo rpm -ihv knox-incubating-{VERSION}.rpm
+This will create a directory `knox-{VERSION}` in your current directory.
+The directory `knox-{VERSION}` will considered your `{GATEWAY_HOME}`
 
 
 ### 6 - Start LDAP embedded in Knox ###
@@ -145,7 +125,7 @@ Knox comes with an LDAP server for demon
 
 ### 7 - Start Knox  ###
 
-The gateway can be started in one of two ways, as java -jar or with a shell script.
+The gateway can be started using the provided shell script.
 
 
 ###### Starting via script
@@ -177,14 +157,14 @@ If for some reason the gateway is stoppe
     cd {GATEWAY_HOME}
     bin/gateway.sh clean
 
-__NOTE: This command will also clear any log output in /var/log/knox so use this with caution.__
+__NOTE: This command will also clear any .out and .err file from the /var/log/knox directory
so use this with caution.__
 
 
 ### 8 - Do Hadoop with Knox
 
 #### Put a file in HDFS via Knox.
 #### CAT a file in HDFS via Knox.
-#### Invoke the LISTSATUS operation on WebHDFS via the gateway.
+#### Invoke the LISTSTATUS operation on WebHDFS via the gateway.
 This will return a directory listing of the root (i.e. /) directory of HDFS.
 
     curl -i -k -u guest:guest-password -X GET \

Modified: knox/trunk/books/0.5.0/service_hbase.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/service_hbase.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/service_hbase.md (original)
+++ knox/trunk/books/0.5.0/service_hbase.md Thu May 22 14:30:00 2014
@@ -17,16 +17,17 @@
 
 ### HBase ###
 
-TODO
+The HBase REST API is provided by the Stargate service for HBase.
+See the HBase Stargate Setup section below for getting started with stargate and Knox with
the Hortonworks Sandbox environment.
 
 #### HBase URL Mapping ####
 
-TODO
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase` |
+| Cluster | `http://{stargate-host}:60080/`                                         |
 
 #### HBase Examples ####
 
-TODO
-
 The examples below illustrate the set of basic operations with HBase instance using Stargate
REST API.
 Use following link to get more more details about HBase/Stargate API: http://wiki.apache.org/hadoop/Hbase/Stargate.
 
@@ -37,7 +38,7 @@ To grant the Read, Write, Create permiss
 
 If you are using a cluster secured with Kerberos you will need to have used `kinit` to authenticate
to the KDC    
 
-### HBase Stargate Setup ###
+#### HBase Stargate Setup ####
 
 #### Launch Stargate ####
 
@@ -67,12 +68,14 @@ If it becomes necessary to restart HBase
     sudo /usr/lib/hbase/bin/hbase-daemon.sh stop rest
     sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop regionserver
     sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop master
+    sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh stop zookeeper
 
     sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start regionserver
     sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start master
+    sudo -u hbase /usr/lib/hbase/bin/hbase-daemon.sh start zookeeper
     sudo /usr/lib/hbase/bin/hbase-daemon.sh start rest -p 60080
 
-### HBase/Stargate client DSL ###
+#### HBase/Stargate client DSL ####
 
 For more details about client DSL usage please follow this [page|https://cwiki.apache.org/confluence/display/KNOX/Client+Usage].
  

Modified: knox/trunk/books/0.5.0/service_hive.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/service_hive.md?rev=1596890&r1=1596889&r2=1596890&view=diff
==============================================================================
--- knox/trunk/books/0.5.0/service_hive.md (original)
+++ knox/trunk/books/0.5.0/service_hive.md Thu May 22 14:30:00 2014
@@ -66,7 +66,7 @@ By default the gateway is configured to 
 #### Hive JDBC URL Mapping ####
 
 | ------- | -------------------------------------------------------------------------------
|
-| Gateway | `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
|
+| Gateway | `jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
|
 | Cluster | `http://{hive-host}:{hive-port}/{hive-path}`                               |
 
 #### Hive Examples ####
@@ -79,27 +79,20 @@ This guide provides detailed examples fo
 2. Make sure Hive Server is running on the correct port.
 3. Make sure Hive Server is running in HTTP mode.
 4. Client side (JDBC):
-    1. Hive JDBC in HTTP mode depends on following libraries to run successfully(must be
in the classpath):
-        * hadoop-common-2.2.0.2.0.6.0-76.jar;
-        * hive-jdbc-0.12.0.2.0.6.0-76.jar;
-        * hive-service-0.12.0.2.0.6.0-76.jar;
-        * libthrift-0.9.0.jar;
-        * httpcore-4.1.4.jar;
-        * httpclient-4.1.3.jar;
-        * hive-common-0.12.0.2.0.6.0-76.jar;
-        * commons-logging-1.1.1.jar;
-        * slf4j-api-1.7.5.jar;
-        * slf4j-log4j12-1.7.5.jar;
-        * log4j-1.2.17.jar;
-        * commons-codec-1.7.jar;
-    2. Import gateway certificate into the default JRE truststore.
-       It is located in the `/lib/security/cacerts`.
-          `keytool -import -alias hadoop.gateway -file hadoop.gateway.cer -keystore <java-home>/lib/security/cacerts`
-       Alternatively you can run your sample with additional parameters:
-          `-Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStore=<path-to-trust-store>
-Djavax.net.ssl.trustStorePassword=<trust-store-password>`
-    3. Connection URL has to be following:
-       `jdbc:hive2://{gateway-host}:{gateway-port}/?hive.server2.transport.mode=https;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
-    4. Look at https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations
for examples.
+     1. Hive JDBC in HTTP mode depends on following minimal libraries set to run successfully(must
be in the classpath):
+         * hive-jdbc-0.13.0.jar;
+         * hive-service-0.13.0.jar;
+         * libthrift-0.9.0.jar;
+         * httpcore-4.2.5.jar;
+         * httpclient-4.2.5.jar;
+         * commons-logging-1.1.3.jar;
+         * commons-codec-1.4.jar;
+         * slf4j-api-1.7.5.jar;
+         * slf4j-log4j12-1.7.5.jar;
+         * log4j-1.2.17.jar;
+     2. Connection URL has to be following:
+        `jdbc:hive2://{gateway-host}:{gateway-port}/;ssl=true;sslTrustStore={gateway-trust-store-path};trustStorePassword={gateway-trust-store-password}?hive.server2.transport.mode=http;hive.server2.thrift.http.path={gateway-path}/{cluster-name}/hive`
+     3. Look at https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DDLOperations
for examples.
        Hint: For testing it would be better to execute `set hive.security.authorization.enabled=false`
as the first statement.
        Hint: Good examples of Hive DDL/DML can be found here http://gettingstarted.hadooponazure.com/hw/hive.html
 
@@ -140,8 +133,10 @@ Sample example for creating new table, l
           String password = user + "-password";
           String gatewayHost = "localhost";
           int gatewayPort = 8443;
+          String trustStore = "/usr/lib/knox/data/security/keystores/gateway.jks";
+          String trustStorePassword = "knoxsecret";
           String contextPath = "gateway/sandbox/hive";
-          String connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s",
gatewayHost, gatewayPort, contextPath );
+          String connectionString = String.format( "jdbc:hive2://%s:%d/;ssl=true;sslTrustStore=%s;trustStorePassword=%s?hive.server2.transport.mode=http;hive.server2.thrift.http.path=/%s",
gatewayHost, gatewayPort, trustStore, trustStorePassword, contextPath );
 
           // load Hive JDBC Driver
           Class.forName( "org.apache.hive.jdbc.HiveDriver" );
@@ -200,18 +195,16 @@ Sample example for creating new table, l
 
 Make sure that GATEWAY_HOME/ext directory contains following libraries for successful execution:
 
-- hadoop-common-2.2.0.2.0.6.0-76.jar;
-- hive-jdbc-0.12.0.2.0.6.0-76.jar;
-- hive-service-0.12.0.2.0.6.0-76.jar;
+- hive-jdbc-0.13.0.jar;
+- hive-service-0.13.0.jar;
 - libthrift-0.9.0.jar;
-- httpcore-4.1.4.jar;
-- httpclient-4.1.3.jar;
-- hive-common-0.12.0.2.0.6.0-76.jar;
-- commons-logging-1.1.1.jar;
+- httpcore-4.2.5.jar;
+- httpclient-4.2.5.jar;
+- commons-logging-1.1.3.jar;
+- commons-codec-1.4.jar;
 - slf4j-api-1.7.5.jar;
 - slf4j-log4j12-1.7.5.jar;
 - log4j-1.2.17.jar;
-- commons-codec-1.7.jar;
 
 There are several ways to execute this sample depending upon your preference.
 
@@ -231,8 +224,10 @@ Each line from the file below will need 
     password = user + "-password";
     gatewayHost = "localhost";
     gatewayPort = 8443;
+    trustStore = "/usr/lib/knox/data/security/keystores/gateway.jks";
+    trustStorePassword = "knoxsecret";
     contextPath = "gateway/sandbox/hive";
-    connectionString = String.format( "jdbc:hive2://%s:%d/?hive.server2.transport.mode=https;hive.server2.thrift.http.path=%s",
gatewayHost, gatewayPort, contextPath );
+    connectionString = String.format( "jdbc:hive2://%s:%d/;ssl=true;sslTrustStore=%s;trustStorePassword=%s?hive.server2.transport.mode=http;hive.server2.thrift.http.path=/%s",
gatewayHost, gatewayPort, trustStore, trustStorePassword, contextPath );
 
     // Load Hive JDBC Driver
     Class.forName( "org.apache.hive.jdbc.HiveDriver" );



Mime
View raw message