knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lmc...@apache.org
Subject svn commit: r1596694 [2/3] - /knox/trunk/books/0.5.0/
Date Wed, 21 May 2014 21:19:07 GMT
Added: knox/trunk/books/0.5.0/config_audit.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_audit.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_audit.md (added)
+++ knox/trunk/books/0.5.0/config_audit.md Wed May 21 21:19:07 2014
@@ -0,0 +1,78 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Audit ###
+
+The Audit facility within the Knox Gateway introduces functionality for tracking actions that are executed by Knox per user's request or that are produced by Knox internal events like topology deploy, etc.
+The Knox Audit module is based on the [Apache log4j](http://logging.apache.org/log4j/1.2/).
+
+#### Configuration needed ####
+
+Out of the box, the Knox Gateway includes preconfigured auditing capabilities. To change its configuration please read following sections.
+
+#### Where audit logs go ####
+
+Audit module is preconfigured to write audit records to the log file `/var/log/knox/gateway-audit.log`.
+
+This behavior can be changed in the `conf/gateway-log4j.properties` file. `log4j.appender.auditfile.*` properties determine this behavior. For detailed information read [Apache log4j](http://logging.apache.org/log4j/1.2/).
+
+#### Audit format ####
+
+Out of the box, the audit record format is defined by org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout.
+Its structure is following:
+
+	EVENT_PUBLISHING_TIME ROOT_REQUEST_ID|PARENT_REQUEST_ID|REQUEST_ID|LOGGER_NAME|TARGET_SERVICE_NAME|USER_NAME|PROXY_USER_NAME|SYSTEM_USER_NAME|ACTION|RESOURCE_TYPE|RESOURCE_NAME|OUTCOME|LOGGING_MESSAGE
+
+The audit record format can be changed by setting `log4j.appender.auditfile.layout` property in `conf/gateway-log4j.properties` to another class that extends org.apache.log4j.Layout or its subclasses.
+
+For detailed information read [Apache log4j](http://logging.apache.org/log4j/1.2/).
+
+##### How to interpret audit log #####
+
+Component | Description
+---------|-----------
+EVENT_PUBLISHING_TIME|Time when audit record was published.
+ROOT_REQUEST_ID|The root request ID if this is a sub-request. Currently it is empty.
+PARENT_REQUEST_ID|The parent request ID if this is a sub-request. Currently it is empty.
+REQUEST_ID|A unique value representing the current, active request. If the current request id value is different from the current parent request id value then the current request id value is moved to the parent request id before it is replaced by the provided request id. If the root request id is not set it will be set with the first non-null value of either the parent request id or the passed request id.
+LOGGER_NAME|The name of the logger
+TARGET_SERVICE_NAME|Name of Hadoop service. Can be empty if audit record is not linked to any Hadoop service, for example, audit record for topology deployment.
+USER_NAME|Name of user that initiated session with Knox
+PROXY_USER_NAME|Mapped user name. For detailed information read #[Identity Assertion].
+SYSTEM_USER_NAME|Currently is empty.
+ACTION|Type of action that was executed. Following actions are defined: authentication, authorization, redeploy, deploy, undeploy, identity-mapping, dispatch, access.
+RESOURCE_TYPE|Type of resource for which action was executed. Following resource types are defined: uri, topology, principal.
+RESOURCE_NAME|Name of resource. For resource of type topology it is name of topology. For resource of type uri it is inbound or dispatch request path. For resource of type principal it is a name of mapped user.
+OUTCOME|Action result type. Following outcomes are defined: success, failure, unavailable.
+LOGGING_MESSAGE| Logging message. Contains additional tracking information.
+
+#### Audit log rotation ####
+
+Audit logging is preconfigured with `org.apache.log4j.DailyRollingFileAppender`.
+[Apache log4j](http://logging.apache.org/log4j/1.2/) contains information about other Appenders.
+
+#### How to change audit level or disable it ####
+
+Audit configuration is stored in the `conf/gateway-log4j.properties` file.
+
+All audit messages are logged at `INFO` level and this behavior can't be changed.
+
+To change audit configuration `log4j.logger.audit*` and `log4j.appender.auditfile*` properties in `conf/gateway-log4j.properties` file should be modified.
+
+Their meaning can be found in [Apache log4j](http://logging.apache.org/log4j/1.2/).
+
+Disabling auditing can be done by decreasing log level for appender.

Added: knox/trunk/books/0.5.0/config_authn.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_authn.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_authn.md (added)
+++ knox/trunk/books/0.5.0/config_authn.md Wed May 21 21:19:07 2014
@@ -0,0 +1,161 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Authentication ###
+
+There are two types of providers supported in Knox for establishing a user's identity:
+
+1. Authentication Providers
+2. Federation Providers
+
+Authentication providers directly accept a user's credentials and validates them against some particular user store. Federation providers, on the other hand, validate a token that has been issued for the user by a trusted Identity Provider (IdP).
+
+The current release of Knox ships with an authentication provider based on the Apache Shiro project and is initially configured for BASIC authentication against an LDAP store. This has been specifically tested against Apache Directory Server and Active Directory.
+
+This section will cover the general approach to leveraging Shiro within the bundled provider including:
+
+1. General mapping of provider config to shiro.ini config
+2. Specific configuration for the bundled BASIC/LDAP configuration
+3. Some tips into what may need to be customized for your environment
+4. How to setup the use of LDAP over SSL or LDAPS
+
+#### General Configuration for Shiro Provider ####
+
+As is described in the configuration section of this document, providers have a name-value based configuration - as is the common pattern in the rest of Hadoop.
+
+The following example shows the format of the configuration for a given provider:
+
+    <provider>
+        <role>authentication</role>
+        <name>ShiroProvider</name>
+        <enabled>true</enabled>
+        <param>
+            <name>{name}</name>
+            <value>{value}</value>
+        </param>
+    </provider>
+
+Conversely, the Shiro provider currently expects a shiro.ini file in the web-inf directory of the cluster specific web application.
+
+The following example illustrates a configuration of the bundled BASIC/LDAP authentication config in a shiro.ini file:
+
+	[urls]
+	/**=authcBasic
+	[main]
+	ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm
+	ldapRealm.contextFactory.authenticationMechanism=simple
+	ldapRealm.contextFactory.url=ldap://localhost:33389
+	ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org
+
+In order to fit into the context of an INI file format, at deployment time we interrogate the paramaters provided in the provider configuration and parse the INI section out of the paramter names. The following provider config illustrates this approach. Notice that the section names in the above shiro.ini match the beginning of the param names that are in the following config:
+
+    <gateway>
+        <provider>
+            <role>authentication</role>
+            <name>ShiroProvider</name>
+            <enabled>true</enabled>
+            <param>
+                <name>main.ldapRealm</name>
+                <value>org.apache.shiro.realm.ldap.JndiLdapRealm</value>
+            </param>
+            <param>
+                <name>main.ldapRealm.userDnTemplate</name>
+                <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+            </param>
+            <param>
+                <name>main.ldapRealm.contextFactory.url</name>
+                <value>ldap://localhost:33389</value>
+            </param>
+            <param>
+                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+                <value>simple</value>
+            </param>
+            <param>
+                <name>urls./**</name>
+                <value>authcBasic</value>
+            </param>
+        </provider>
+
+This happens to be the way that we are currently configuring Shiro for BASIC/LDAP authentication. This same config approach may be used to achieve other authentication mechanisms or variations on this one. We however have not tested additional uses for it for this release.
+
+#### LDAP Configuration ####
+
+This section discusses the LDAP configuration used above for the Shiro Provider. Some of these configuration elements will need to be customized to reflect your deployment environment.
+
+**main.ldapRealm** - this element indicates the fully qualified classname of the Shiro realm to be used in authenticating the user. The classname provided by default in the sample is the `org.apache.shiro.realm.ldap.JndiLdapRealm` this implementation provides us with the ability to authenticate but by default has authorization disabled. In order to provide authorization - which is seen by Shiro as dependent on an LDAP schema that is specific to each organization - an extension of JndiLdapRealm is generally used to override and implement the doGetAuhtorizationInfo method. In this particular release we are providing a simple authorization provider that can be used along with the Shiro authentication provider.
+
+**main.ldapRealm.userDnTemplate** - in order to bind a simple username to an LDAP server that generally requires a full distinguished name (DN), we must provide the template into which the simple username will be inserted. This template allows for the creation of a DN by injecting the simple username into the common name (CN) portion of the DN. **This element will need to be customized to reflect your deployment environment.** The template provided in the sample is only an example and is valid only within the LDAP schema distributed with Knox and is represented by the users.ldif file in the {GATEWAY_HOME}conf directory.
+
+**main.ldapRealm.contextFactory.url** - this element is the URL that represents the host and port of LDAP server. It also includes the scheme of the protocol to use. This may be either ldap or ldaps depending on whether you are communicating with the LDAP over SSL (higly recommended). **This element will need to be cusomized to reflect your deployment environment.**.
+
+**main.ldapRealm.contextFactory.authenticationMechanism** - this element indicates the type of authentication that should be performed against the LDAP server. The current default value is `simple` which indicates a simple bind operation. This element should not need to be modified and no mechanism other than a simple bind has been tested for this particular release.
+
+**urls./**** - this element represents a single URL_Ant_Path_Expression and the value the Shiro filter chain to apply to it. This particular sample indicates that all paths into the application have the same Shiro filter chain applied. The paths are relative to the application context path. The use of the value `authcBasic` here indicates that BASIC authentication is expected for every path into the application. Adding an additional Shiro filter to that chain for validating that the request isSecure() and over SSL can be achieved by changing the value to `ssl, authcBasic`. It is not likely that you need to change this element for your environment.
+
+#### Active Directory - Special Note ####
+
+You would use LDAP configuration as documented above to authenticate against Active Directory as well.
+
+Some Active Directory specifc things to keep in mind:
+
+Typical AD main.ldapRealm.userDnTemplate value looks slightly different, such as
+    cn={0},cn=users,DC=lab,DC=sample,dc=com
+
+Please compare this with a typical Apache DS main.ldapRealm.userDnTemplate value and make note of the difference.
+    uid={0},ou=people,dc=hadoop,dc=apache,dc=org
+
+If your AD is configured to authenticate based on just the cn and password and does not require user DN, you do not have to specify value for  main.ldapRealm.userDnTemplate.
+
+
+#### LDAP over SSL (LDAPS) Configuration ####
+In order to communicate with your LDAP server over SSL (again, highly recommended), you will need to modify the topology file in a couple ways and possibly provision some keying material.
+
+1. **main.ldapRealm.contextFactory.url** must be changed to have the `ldaps` protocol scheme and the port must be the SSL listener port on your LDAP server.
+2. Identity certificate (keypair) provisioned to LDAP server - your LDAP server specific documentation should indicate what is requried for providing a cert or keypair to represent the LDAP server identity to connecting clients.
+3. Trusting the LDAP Server's public key - if the LDAP Server's identity certificate is issued by a well known and trusted certificate authority and is already represented in the JRE's cacerts truststore then you don't need to do anything for trusting the LDAP server's cert. If, however, the cert is selfsigned or issued by an untrusted authority you will need to either add it to the cacerts keystore or to another truststore that you may direct Knox to utilize through a system property.
+
+#### Session Configuration ####
+
+Knox maps each cluster topology to a web application and leverages standard JavaEE session management.
+
+To configure session idle timeout for the topology, please specify value of parameter sessionTimeout for ShiroProvider in your topology file.  If you do not specify the value for this parameter, it defaults to 30minutes.
+
+The definition would look like the following in the topoloogy file:
+
+    ...
+    <provider>
+        <role>authentication</role>
+        <name>ShiroProvider</name>
+        <enabled>true</enabled>
+        <param>
+            <!--
+            Session timeout in minutes. This is really idle timeout.
+            Defaults to 30 minutes, if the property value is not defined.
+            Current client authentication will expire if client idles
+            continuously for more than this value
+            -->
+            <name>sessionTimeout</name>
+            <value>30</value>
+        </param>
+    <provider>
+    ...
+
+
+At present, ShiroProvider in Knox leverages JavaEE session to maintain authentication state for a user across requests using JSESSIONID cookie.  So, a clieent that authenticated with Knox could pass the JSESSIONID cookie with repeated requests as long as the session has not timed out instead of submitting userid/password with every request.  Presenting a valid session cookie in place of userid/password would also perform better as additional credential store lookups are avoided.
+
+
+

Added: knox/trunk/books/0.5.0/config_authz.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_authz.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_authz.md (added)
+++ knox/trunk/books/0.5.0/config_authz.md Wed May 21 21:19:07 2014
@@ -0,0 +1,349 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Authorization ###
+
+#### Service Level Authorization ####
+
+The Knox Gateway has an out-of-the-box authorization provider that allows administrators to restrict access to the individual services within a Hadoop cluster.
+
+This provider utilizes a simple and familiar pattern of using ACLs to protect Hadoop resources by specifying users, groups and ip addresses that are permitted access.
+
+Note: In the examples below \{serviceName\} represents a real service name (e.g. WEBHDFS) and would be replaced with these values in an actual configuration.
+
+##### Usecases #####
+
+###### USECASE-1: Restrict access to specific Hadoop services to specific Users
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;*;*</value>
+    </param>
+
+###### USECASE-2: Restrict access to specific Hadoop services to specific Groups
+
+    <param>
+        <name>{serviceName}.acls</name>
+        <value>*;admins;*</value>
+    </param>
+
+###### USECASE-3: Restrict access to specific Hadoop services to specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>*;*;127.0.0.1</value>
+    </param>
+
+###### USECASE-4: Restrict access to specific Hadoop services to specific Users OR users within specific Groups
+
+    <param>
+        <name>{serviceName}.acl.mode</name>
+        <value>OR</value>
+    </param>
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;admin;*</value>
+    </param>
+
+###### USECASE-5: Restrict access to specific Hadoop services to specific Users OR users from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl.mode</name>
+        <value>OR</value>
+    </param>
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;*;127.0.0.1</value>
+    </param>
+
+###### USECASE-6: Restrict access to specific Hadoop services to users within specific Groups OR from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl.mode</name>
+        <value>OR</value>
+    </param>
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>*;admin;127.0.0.1</value>
+    </param>
+
+###### USECASE-7: Restrict access to specific Hadoop services to specific Users OR users within specific Groups OR from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl.mode</name>
+        <value>OR</value>
+    </param>
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;admin;127.0.0.1</value>
+    </param>
+
+###### USECASE-8: Restrict access to specific Hadoop services to specific Users AND users within specific Groups
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;admin;*</value>
+    </param>
+
+###### USECASE-9: Restrict access to specific Hadoop services to specific Users AND users from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;*;127.0.0.1</value>
+    </param>
+
+###### USECASE-10: Restrict access to specific Hadoop services to users within specific Groups AND from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>*;admins;127.0.0.1</value>
+    </param>
+
+###### USECASE-11: Restrict access to specific Hadoop services to specific Users AND users within specific Groups AND from specific Remote IPs
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>guest;admins;127.0.0.1</value>
+    </param>
+
+#### Configuration ####
+
+ACLs are bound to services within the topology descriptors by introducing the authorization provider with configuration like:
+
+    <provider>
+        <role>authorization</role>
+        <name>AclsAuthz</name>
+        <enabled>true</enabled>
+    </provider>
+
+The above configuration enables the authorization provider but does not indicate any ACLs yet and therefore there is no restriction to accessing the Hadoop services. In order to indicate the resources to be protected and the specific users, groups or ip's to grant access, we need to provide parameters like the following:
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>username[,*|username...];group[,*|group...];ipaddr[,*|ipaddr...]</value>
+    </param>
+    
+where `{serviceName}` would need to be the name of a configured Hadoop service within the topology.
+
+NOTE: ipaddr is unique among the parts of the ACL in that you are able to specify a wildcard within an ipaddr to indicate that the remote address must being with the String prior to the asterisk within the ipaddr acl. For instance:
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>*;*;192.168.*</value>
+    </param>
+    
+This indicates that the request must come from an IP address that begins with '192.168.' in order to be granted access.
+
+Note also that configuration without any ACLs defined is equivalent to:
+
+    <param>
+        <name>{serviceName}.acl</name>
+        <value>*;*;*</value>
+    </param>
+
+meaning: all users, groups and IPs have access.
+Each of the elements of the acl param support multiple values via comma separated list and the `*` wildcard to match any.
+
+For instance:
+
+    <param>
+        <name>webhdfs.acl</name>
+        <value>hdfs;admin;127.0.0.2,127.0.0.3</value>
+    </param>
+
+this configuration indicates that ALL of the following are satisfied:
+
+1. the user "hdfs" has access AND
+2. users in the group "admin" have access AND
+3. any authenticated user from either 127.0.0.2 or 127.0.0.3 will have access
+
+This allows us to craft policy that restricts the members of a large group to a subset that should have access.
+The user being removed from the group will allow access to be denied even though their username may have been in the ACL.
+
+An additional configuration element may be used to alter the processing of the ACL to be OR instead of the default AND behavior:
+
+    <param>
+        <name>{serviceName}.acl.mode</name>
+        <value>OR</value>
+    </param>
+
+this processing behavior requires that the effective user satisfy one of the parts of the ACL definition in order to be granted access.
+For instance:
+
+    <param>
+        <name>webhdfs.acl</name>
+        <value>hdfs,guest;admin;127.0.0.2,127.0.0.3</value>
+    </param>
+
+You may also set the ACL processing mode at the top level for the topology. This essentially sets the default for the managed cluster.
+It may then be overridden at the service level as well.
+
+    <param>
+        <name>acl.mode</name>
+        <value>OR</value>
+    </param>
+
+this configuration indicates that ONE of the following must be satisfied to be granted access:
+
+1. the user is "hdfs" or "guest" OR
+2. the user is in "admin" group OR
+3. the request is coming from 127.0.0.2 or 127.0.0.3
+
+#### Other Related Configuration ####
+
+The principal mapping aspect of the identity assertion provider is important to understand in order to fully utilize the authorization features of this provider.
+
+This feature allows us to map the authenticated principal to a runas or impersonated principal to be asserted to the Hadoop services in the backend.
+When a principal mapping is defined that results in an impersonated principal being created the impersonated principal is then the effective principal.
+If there is no mapping to another principal then the authenticated or primary principal is then the effective principal.
+Principal mapping has actually been available in the identity assertion provider from the beginning of Knox and is documented fully in the Identity Assertion section of this guide.
+
+    <param>
+        <name>principal.mapping</name>
+        <value>{primaryPrincipal}[,...]={impersonatedPrincipal}[;...]</value>
+    </param>
+
+For instance:
+
+    <param>
+        <name>principal.mapping</name>
+        <value>guest=hdfs</value>
+    </param>
+
+In addition, we allow the administrator to map groups to effective principals. This is done through another param within the identity assertion provider:
+
+    <param>
+        <name>group.principal.mapping</name>
+        <value>{userName[,*|userName...]}={groupName[,groupName...]}[,...]</value>
+    </param>
+
+For instance:
+
+    <param>
+        <name>group.principal.mapping</name>
+        <value>*=users;hdfs=admin</value>
+    </param>
+
+this configuration indicates that all (*) authenticated users are members of the "users" group and that user "hdfs" is a member of the admin group. Group principal mapping has been added along with the authorization provider described in this document.
+
+For more information on principal and group principal mapping see the Identity Assertion section of this guide.
+
+These additional mapping capabilities are used together with the authorization ACL policy.
+An example of a full topology that illustrates these together is below.
+
+    <topology>
+        <gateway>
+            <provider>
+                <role>authentication</role>
+                <name>ShiroProvider</name>
+                <enabled>true</enabled>
+                <param>
+                    <name>main.ldapRealm</name>
+                    <value>org.apache.shiro.realm.ldap.JndiLdapRealm</value>
+                </param>
+                <param>
+                    <name>main.ldapRealm.userDnTemplate</name>
+                    <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+                </param>
+                <param>
+                    <name>main.ldapRealm.contextFactory.url</name>
+                    <value>ldap://localhost:33389</value>
+                </param>
+                <param>
+                    <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+                    <value>simple</value>
+                </param>
+                <param>
+                    <name>urls./**</name>
+                    <value>authcBasic</value>
+                </param>
+            </provider>
+            <provider>
+                <role>identity-assertion</role>
+                <name>Pseudo</name>
+                <enabled>true</enabled>
+                <param>
+                    <name>principal.mapping</name>
+                    <value>guest=hdfs;</value>
+                </param>
+                <param>
+                    <name>group.principal.mapping</name>
+                    <value>*=users;hdfs=admin</value>
+                </param>
+            </provider>
+            <provider>
+                <role>authorization</role>
+                <name>AclsAuthz</name>
+                <enabled>true</enabled>
+                <param>
+                    <name>acl.mode</name>
+                    <value>OR</value>
+                </param>
+                <param>
+                    <name>webhdfs.acl.mode</name>
+                    <value>AND</value>
+                </param>
+                <param>
+                    <name>webhdfs.acl</name>
+                    <value>hdfs;admin;127.0.0.2,127.0.0.3</value>
+                </param>
+                <param>
+                    <name>webhcat.acl</name>
+                    <value>hdfs;admin;127.0.0.2,127.0.0.3</value>
+                </param>
+            </provider>
+            <provider>
+                <role>hostmap</role>
+                <name>static</name>
+                <enabled>true</enabled>
+                <param>
+                    <name>localhost</name>
+                    <value>sandbox,sandbox.hortonworks.com</value>
+                </param>
+            </provider>
+        </gateway>
+
+		<service>
+        	<role>JOBTRACKER</role>
+        	<url>rpc://localhost:8050</url>
+    	</service>
+
+    	<service>
+        	<role>WEBHDFS</role>
+        	<url>http://localhost:50070/webhdfs</url>
+    	</service>
+
+    	<service>
+        	<role>WEBHCAT</role>
+        	<url>http://localhost:50111/templeton</url>
+    	</service>
+
+    	<service>
+        	<role>OOZIE</role>
+        	<url>http://localhost:11000/oozie</url>
+    	</service>
+
+    	<service>
+        	<role>WEBHBASE</role>
+        	<url>http://localhost:60080</url>
+    	</service>
+
+    	<service>
+        	<role>HIVE</role>
+        	<url>http://localhost:10001/cliservice</url>
+    	</service>
+    </topology>

Added: knox/trunk/books/0.5.0/config_ha.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_ha.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_ha.md (added)
+++ knox/trunk/books/0.5.0/config_ha.md Wed May 21 21:19:07 2014
@@ -0,0 +1,124 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### High Availability ###
+
+#### Configure Knox instances ####
+
+All Knox instances must be synced to use the same topologies credentials keystores.
+These files are located under {GATEWAY_HOME}/conf/security/keystores/{TOPOLOGY_NAME}-credentials.jceks.
+They are generated after the first topology deployment.
+Currently these files can be synced just manually. There is no automation tool.
+Here are the steps to sync topologies credentials keystores:
+
+1. Choose Knox instance that will be the source for topologies credentials keystores. Let's call it keystores master
+1. Replace topologies credentials keystores in the other Knox instance with topologies credentials keystores from keystores master
+1. Restart Knox instances
+
+#### High Availability with Apache HTTP Server + mod_proxy + mod_proxy_balancer ####
+
+##### 1 - Requirements #####
+
+###### openssl-devel ######
+
+openssl-devel is required for Apache Module mod_ssl.
+
+    sudo yum install openssl-devel
+
+###### Apache HTTP Server ######
+
+Apache HTTP Server 2.4.6 or later is required. See this document for installing and setting up Apache HTTP Server: http://httpd.apache.org/docs/2.4/install.html
+
+Hint: pass --enable-ssl to ./configure command to enable Apache Module mod_ssl generation.
+
+###### Apache Module mod_proxy ######
+
+See this document for setting up Apache Module mod_proxy: http://httpd.apache.org/docs/2.4/mod/mod_proxy.html
+
+###### Apache Module mod_proxy_balancer ######
+
+See this document for setting up Apache Module mod_proxy_balancer: http://httpd.apache.org/docs/2.4/mod/mod_proxy_balancer.html
+
+###### Apache Module mod_ssl ######
+
+See this document for setting up Apache Module mod_ssl: http://httpd.apache.org/docs/2.4/mod/mod_ssl.html
+
+##### 2 - Configuration example #####
+
+###### Generate certificate for Apache HTTP Server ######
+
+See this document for an example: http://www.akadia.com/services/ssh_test_certificate.html
+
+By convention, Apache HTTP Server and Knox certificates are put into /etc/apache2/ssl/ folder.
+
+###### Update Apache HTTP Server configuration file ######
+
+This file is located under {APACHE_HOME}/conf/httpd.conf.
+
+Following directives have to be added or uncommented in the configuration file:
+
+* LoadModule proxy_module modules/mod_proxy.so
+* LoadModule proxy_http_module modules/mod_proxy_http.so
+* LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
+* LoadModule ssl_module modules/mod_ssl.so
+* LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
+* LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
+* LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
+* LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
+* LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
+
+Also following lines have to be added to file. Replace placeholders (${...}) with real data:
+
+    Listen 443
+    <VirtualHost *:443>
+       SSLEngine On
+       SSLProxyEngine On
+       SSLCertificateFile ${PATH_TO_CERTICICATE_FILE}
+       SSLCertificateKeyFile ${PATH_TO_CERTICICATE_KEY_FILE}
+       SSLProxyCACertificateFile ${PATH_TO_PROXY_CA_CERTICICATE_FILE}
+
+       ProxyRequests Off
+       ProxyPreserveHost Off
+
+       Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
+       <Proxy balancer://mycluster>
+         BalancerMember ${HOST_#1} route=1
+         BalancerMember ${HOST_#2} route=2
+         ...
+         BalancerMember ${HOST_#N} route=N
+
+         ProxySet failontimeout=On lbmethod=${LB_METHOD} stickysession=ROUTEID 
+       </Proxy>
+
+       ProxyPass / balancer://mycluster/
+       ProxyPassReverse / balancer://mycluster/
+    </VirtualHost>
+
+Note:
+
+* SSLProxyEngine enables SSL between Apache HTTP Server and Knox instances;
+* SSLCertificateFile and SSLCertificateKeyFile have to point to certificate data of Apache HTTP Server. User will use this certificate for communications with Apache HTTP Server;
+* SSLProxyCACertificateFile has to point to Knox certificates.
+
+###### Start/stop Apache HTTP Server ######
+
+    APACHE_HOME/bin/apachectl -k start
+    APACHE_HOME/bin/apachectl -k stop
+
+###### Verify ######
+
+Use Knox samples.

Added: knox/trunk/books/0.5.0/config_id_assertion.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_id_assertion.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_id_assertion.md (added)
+++ knox/trunk/books/0.5.0/config_id_assertion.md Wed May 21 21:19:07 2014
@@ -0,0 +1,100 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Identity Assertion ###
+The identity assertion provider within Knox plays the critical role of communicating the identity principal to be used within the Hadoop cluster to represent the identity that has been authenticated at the gateway.
+
+The general responsibilities of the identity assertion provider is to interrogate the current Java Subject that has been established by the authentication or federation provider and:
+
+1. determine whether it matches any principal mapping rules and apply them appropriately
+2. determine whether it matches any group principal mapping rules and apply them
+3. if it is determined that the principal will be impersonating another through a principal mapping rule then a Subject.doAS is required in order for providers farther downstream can determine the appropriate effective principal name and groups for the user
+
+The following configuration is required for asserting the users identity to the Hadoop cluster using Pseudo or Simple "authentication".
+
+    <provider>
+        <role>identity-assertion</role>
+        <name>Pseudo</name>
+        <enabled>true</enabled>
+    </provider>
+
+This particular configuration indicates that the Pseudo identity assertion provider is enabled and that there are no principal mapping rules to apply to identities flowing from the authentication in the gateway to the backend Hadoop cluster services. The primary principal of the current subject will therefore be asserted via a query paramter or as a form parameter - ie. ?user.name={primaryPrincipal}
+
+    <provider>
+        <role>identity-assertion</role>
+        <name>Pseudo</name>
+        <enabled>true</enabled>
+        <param>
+            <name>principal.mapping</name>
+            <value>guest=hdfs;</value>
+        </param>
+        <param>
+            <name>group.principal.mapping</name>
+            <value>*=users;hdfs=admin</value>
+        </param>
+    </provider>
+
+This configuration identifies the same identity assertion provider but does provide principal and group mapping rules. In this case, when a user is authenticated as "guest" his identity is actually asserted to the Hadoop cluster as "hdfs". In addition, since there are group principal mappings defined, he will also be considered as a member of the groups "users" and "admin". In this particular example the wildcard "*" is used to indicate that all authenticated users need to be considered members of the "users" group and that only the user "hdfs" is mapped to be a member of the "admin" group.
+
+**NOTE: These group memberships are currently only meaningful for Service Level Authorization using the AclsAuthorization provider. The groups are not currently asserted to the Hadoop cluster at this time. See the Authorization section within this guide to see how this is used.**
+
+The principal mapping aspect of the identity assertion provider is important to understand in order to fully utilize the authorization features of this provider.
+
+This feature allows us to map the authenticated principal to a runas or impersonated principal to be asserted to the Hadoop services in the backend.
+
+When a principal mapping is defined that results in an impersonated principal being created the impersonated principal is then the effective principal.
+
+If there is no mapping to another principal then the authenticated or primary principal is then the effective principal.
+
+#### Principal Mapping ####
+
+    <param>
+        <name>principal.mapping</name>
+        <value>{primaryPrincipal}[,...]={impersonatedPrincipal}[;...]</value>
+    </param>
+
+For instance:
+
+    <param>
+        <name>principal.mapping</name>
+        <value>guest=hdfs</value>
+    </param>
+
+For multiple mappings:
+
+    <param>
+        <name>principal.mapping</name>
+        <value>guest,alice=hdfs;mary=alice2</value>
+    </param>
+
+#### Group Principal Mapping ####
+
+    <param>
+        <name>group.principal.mapping</name>
+        <value>{userName[,*|userName...]}={groupName[,groupName...]}[,...]</value>
+    </param>
+
+For instance:
+
+    <param>
+        <name>group.principal.mapping</name>
+        <value>*=users;hdfs=admin</value>
+    </param>
+
+this configuration indicates that all (*) authenticated users are members of the "users" group and that user "hdfs" is a member of the admin group. Group principal mapping has been added along with the authorization provider described in this document.
+
+

Added: knox/trunk/books/0.5.0/config_kerberos.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_kerberos.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_kerberos.md (added)
+++ knox/trunk/books/0.5.0/config_kerberos.md Wed May 21 21:19:07 2014
@@ -0,0 +1,119 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Secure Clusters ###
+
+See these documents for setting up a secure Hadoop cluster
+http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuration_in_Secure_Mode
+http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14.html
+
+Once you have a Hadoop cluster that is using Kerberos for authentication, you have to do the following to configure Knox to work with that cluster.
+
+#### Create Unix account for Knox on Hadoop master nodes ####
+
+    useradd -g hadoop knox
+
+#### Create Kerberos principal, keytab for Knox ####
+
+One way of doing this, assuming your KDC realm is EXAMPLE.COM, is to ssh into your host running KDC and execute `kadmin.local`
+That will result in an interactive session in which you can execute commands.
+
+ssh into your host running KDC
+
+    kadmin.local
+    add_principal -randkey knox/knox@EXAMPLE.COM
+    ktadd -norandkey -k /etc/security/keytabs/knox.service.keytab
+    ktadd -k /etc/security/keytabs/knox.service.keytab -norandkey knox/knox@EXAMPLE.COM
+    exit
+
+#### Grant Proxy privileges for Knox user in `core-site.xml` on Hadoop master nodes ####
+
+Update `core-site.xml` and add the following lines towards the end of the file.
+
+Replace FQDN_OF_KNOX_HOST with the fully qualified domain name of the host running the gateway.
+You can usually find this by running `hostname -f` on that host.
+
+You could use * for local developer testing if Knox host does not have static IP.
+
+    <property>
+        <name>webhcat.proxyuser.knox.groups</name>
+        <value>users</value>
+    </property>
+    <property>
+        <name>webhcat.proxyuser.knox.hosts</name>
+        <value>FQDN_OF_KNOX_HOST</value>
+    </property>
+
+#### Grant proxy privilege for Knox in `webhcat-stie.xml` on Hadoop master nodes ####
+
+Update `webhcat-site.xml` and add the following lines towards the end of the file.
+
+Replace FQDN_OF_KNOX_HOST with right value in your cluster.
+You could use * for local developer testing if Knox host does not have static IP.
+
+    <property>
+        <name>hadoop.proxyuser.knox.groups</name>
+        <value>users</value>
+    </property>
+    <property>
+        <name>hadoop.proxyuser.knox.hosts</name>
+        <value>FQDN_OF_KNOX_HOST</value>
+    </property>
+
+#### Grant proxy privilege for Knox in `oozie-stie.xml` on Oozie host ####
+
+Update `oozie-site.xml` and add the following lines towards the end of the file.
+
+Replace FQDN_OF_KNOX_HOST with right value in your cluster.
+You could use * for local developer testing if Knox host does not have static IP.
+
+    <property>
+       <name>oozie.service.ProxyUserService.proxyuser.knox.groups</name>
+       <value>users</value>
+    </property>
+    <property>
+       <name>oozie.service.ProxyUserService.proxyuser.knox.hosts</name>
+       <value>FQDN_OF_KNOX_HOST</value>
+    </property>
+
+#### Copy knox keytab to Knox host ####
+
+Add unix account for the knox user on Knox host
+
+    useradd -g hadoop knox
+
+Copy knox.service.keytab created on KDC host on to your Knox host /etc/knox/conf/knox.service.keytab
+
+    chown knox knox.service.keytab
+    chmod 400 knox.service.keytab
+
+#### Update krb5.conf at /etc/knox/conf/krb5.conf on Knox host ####
+
+You could copy the `templates/krb5.conf` file provided in the Knox binary download and customize it to suit your cluster.
+
+#### Update `krb5JAASLogin.conf` at `/etc/knox/conf/krb5JAASLogin.conf` on Knox host ####
+
+You could copy the `templates/krb5JAASLogin.conf` file provided in the Knox binary download and customize it to suit your cluster.
+
+#### Update `gateway-site.xml` on Knox host on Knox host ####
+
+Update `conf/gateway-site.xml` in your Knox installation and set the value of `gateway.hadoop.kerberos.secured` to true.
+
+#### Restart Knox ####
+
+After you do the above configurations and restart Knox, Knox would use SPNego to authenticate with Hadoop services and Oozie.
+There is no change in the way you make calls to Knox whether you use Curl or Knox DSL.

Added: knox/trunk/books/0.5.0/config_ldap_group_lookup.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_ldap_group_lookup.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_ldap_group_lookup.md (added)
+++ knox/trunk/books/0.5.0/config_ldap_group_lookup.md Wed May 21 21:19:07 2014
@@ -0,0 +1,212 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### LDAPGroupLookup ###
+
+Knox can be configured to look up LDAP groups that the authenticated user belong to.
+Knox can look up both Static LDAP Groups and Dynamic LDAP Groups.
+The looked up groups are populated as Principal(s) in the Java Subject of authenticated user.
+Therefore service authorization rules can be defined in terms of LDAPGroups looked up from LDAP directory.
+
+To look up LDAPGroups of autheticated user from LDAP, you have to use org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm in Shiro configuration.
+
+Please see below a sample Shiro configuration snippet from a topology file that was tested looking LDAPGroups.
+
+        <provider>
+            <role>authentication</role>
+            <name>ShiroProvider</name>
+            <enabled>true</enabled>
+            <!-- 
+            session timeout in minutes,  this is really idle timeout,
+            defaults to 30mins, if the property value is not defined,, 
+            current client authentication would expire if client idles contiuosly for more than this value
+            -->
+            <!-- defaults to: 30 minutes
+            <param>
+                <name>sessionTimeout</name>
+                <value>30</value>
+            </param>
+            -->
+
+            <!--
+              Use single KnoxLdapRealm to do authentication and ldap group look up
+            -->
+            <param>
+              <name>main.ldapRealm</name>
+              <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
+            </param>
+            <param>
+              <name>main.ldapGroupContextFactory</name>
+              <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
+            </param>
+            <param>
+              <name>main.ldapRealm.contextFactory</name>
+              <value>$ldapGroupContextFactory</value>
+            </param>
+            <!-- defaults to: simple
+            <param>
+              <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
+              <value>simple</value>
+            </param>
+            -->
+            <param>
+              <name>main.ldapRealm.contextFactory.url</name>
+              <value>ldap://localhost:33389</value>
+            </param>
+            <param>
+              <name>main.ldapRealm.userDnTemplate</name>
+              <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+            </param>
+
+            <param>
+              <name>main.ldapRealm.authorizationEnabled</name>
+              <!-- defaults to: false -->
+              <value>true</value>
+            </param>
+            <!-- defaults to: simple
+            <param>
+              <name>main.ldapRealm.contextFactory.systemAuthenticationMechanism</name>
+              <value>simple</value>
+            </param>
+            -->
+            <param>
+              <name>main.ldapRealm.searchBase</name>
+              <value>ou=groups,dc=hadoop,dc=apache,dc=org</value>
+            </param>
+            <!-- defaults to: groupOfNames
+            <param>
+              <name>main.ldapRealm.groupObjectClass</name>
+              <value>groupOfNames</value>
+            </param>
+            -->
+            <!-- defaults to: member
+            <param>
+              <name>main.ldapRealm.memberAttribute</name>
+              <value>member</value>
+            </param>
+            -->
+            <param>
+              <name>main.ldapRealm.memberAttributeValueTemplate</name>
+              <value>cn={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
+            </param>
+            <param>
+              <name>main.ldapRealm.contextFactory.systemUsername</name>
+              <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value>
+            </param>
+            <param>
+              <name>main.ldapRealm.contextFactory.systemPassword</name>
+              <value>${ALIAS=ldcSystemPassword}</value>
+            </param>
+
+            <param>
+              <name>urls./**</name> 
+              <value>authcBasic</value>
+            </param>
+
+        </provider>
+
+The configuration shown above would look up Static LDAP groups of authenticated user and populate the group principals in the Java Subject corresponding to authenticated user.
+
+If you want to look up Dynamic LDAP Groups instead of Static LDAP Groups, you would have to speciify groupObjectClass and memberAttribute params as shown below:
+
+           <param>
+              <name>main.ldapRealm.groupObjectClass</name>
+              <value>groupOfUrls</value>
+            </param>
+            <param>
+              <name>main.ldapRealm.memberAttribute</name>
+              <value>memberUrl</value>
+            </param>
+
+### Template topology files and LDIF files to try out LDAP Group Look up ###
+
+Knox bundles some template topology files and ldif files that you can use to try and test LDAP Group Lookup and associated authorization acls.
+All these template files are located under {GATEWAY_HOME}/templates.
+
+
+#### LDAP Static Group Lookup Templates, authentication and group lookup from the same directory ####
+
+topology file: sandbox.knoxrealm1.xml
+ldif file    : users.ldapgroups.ldif
+
+To try this out
+
+cd {GATEWAY_HOME}
+cp templates/sandbox.knoxrealm1.xml deployments/sandbox.xml
+cp templates/users.ldapgroups.ldif conf/users.ldif
+java -jar bin/ldap.jar conf
+java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master
+
+Following call to WebHDFS should report HTTP/1.1 401 Unauthorized
+As guest is not a member of group "analyst", authorization prvoider states user should be member of group "analyst"
+
+curl  -i -v  -k -u guest:guest-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+
+Following call to WebHDFS should report: {"Path":"/user/sam"}
+As sam is a member of group "analyst", authorization prvoider states user should be member of group "analyst"
+
+curl  -i -v  -k -u sam:sam-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+
+
+#### LDAP Static Group Lookup Templates, authentication and group lookup from different  directories ####
+
+topology file: sandbox.knoxrealm2.xml
+ldif file    : users.ldapgroups.ldif
+
+To try this out
+
+cd {GATEWAY_HOME}
+cp templates/sandbox.knoxrealm2.xml deployments/sandbox.xml
+cp templates/users.ldapgroups.ldif conf/users.ldif
+java -jar bin/ldap.jar conf
+java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master
+
+Following call to WebHDFS should report HTTP/1.1 401 Unauthorized
+As guest is not a member of group "analyst", authorization prvoider states user should be member of group "analyst"
+
+curl  -i -v  -k -u guest:guest-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+
+Following call to WebHDFS should report: {"Path":"/user/sam"}
+As sam is a member of group "analyst", authorization prvoider states user should be member of group "analyst"
+
+curl  -i -v  -k -u sam:sam-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+
+#### LDAP Dynamic Group Lookup Templates, authentication and dynamic group lookup from same  directory ####
+
+topology file: sandbox.knoxrealmdg.xml
+ldif file    : users.ldapdynamicgroups.ldif
+
+To try this out
+
+cd {GATEWAY_HOME}
+cp templates/sandbox.knoxrealmdg.xml deployments/sandbox.xml
+cp templates/users.ldapdynamicgroups.ldif conf/users.ldif
+java -jar bin/ldap.jar conf
+java -Dsandbox.ldcSystemPassword=guest-password -jar bin/gateway.jar -persist-master
+
+Please note that user.ldapdynamicgroups.ldif also loads ncessary schema to create dynamic groups in Apache DS.
+
+Following call to WebHDFS should report HTTP/1.1 401 Unauthorized
+As guest is not a member of dynamic group "directors", authorization prvoider states user should be member of group "directors"
+
+curl  -i -v  -k -u guest:guest-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+
+Following call to WebHDFS should report: {"Path":"/user/bob"}
+As bob is a member of dynamic group "directors", authorization prvoider states user should be member of group "directors"
+
+curl  -i -v  -k -u sam:sam-password  -X GET https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY
+

Added: knox/trunk/books/0.5.0/config_preauth_sso_provider.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_preauth_sso_provider.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_preauth_sso_provider.md (added)
+++ knox/trunk/books/0.5.0/config_preauth_sso_provider.md Wed May 21 21:19:07 2014
@@ -0,0 +1,83 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Preauthenticated SSO Provider ###
+
+A number of SSO solutions provide mechanisms for federating an authenticated identity across applications. These mechanisms are at times simple HTTP Header type tokens that can be used to propagate the identity across process boundaries.
+
+Knox Gateway needs a pluggable mechanism for consuming these tokens and federating the asserted identity through an interaction with the Hadoop cluster. 
+
+**CAUTION: The use of this provider requires that proper network security and identity provider configuration and deployment does not allow requests directly to the Knox gateway. Otherwise, this provider will leave the gateway exposed to identity spoofing.**
+
+#### Configuration ####
+##### Overview #####
+The HeaderPreAuth provider is configured within the topology file and has a minimal configuration that assumes SM_USER for CA SiteMinder. The following example is the bare minimum configuration for SiteMinder (with no IP address validation).
+
+	<provider>
+      <role>federation</role>
+      <name>HeaderPreAuth</name>
+      <enabled>true</enabled>
+    </provider>
+
+The following table describes the configuration options for the web app security provider:
+
+##### Descriptions #####
+
+Name | Description | Default
+---------|-----------
+preauth.validation.method|Optional parameter that indicates the type of trust validation to perform on incoming requests. Possible values are: null, preauth.ip.validation (others will be added in future releases). Failure results in a 403 forbidden HTTP status response.|null - which means no validation will be performed and that we are assuming that the network security and external authentication system is sufficient.  
+preauth.ip.addresses|Optional parameter that indicates the list of trusted ip addresses. When preauth.ip.validation is indicated as the validation method this parameter must be provided to indicate the trusted ip address set. Wildcarded IPs may be used to indicate subnet level trust. ie. 127.0.*|null - which means that no validation will be performed.
+preauth.custom.header|Required parameter for indicating a custom header to use for extracting the preauthenticated principal. The value extracted from this header is utilized as the PrimaryPrincipal within the established Subject. An incoming request that is missing the configured header will be refused with a 401 unauthorized HTTP status.|SM_USER for SiteMinder usecase
+preauth.custom.group.header|Optional parameter for indicating a HTTP header name that contains a comma separated list of groups. These are added to the authenticated Subject as group principals. A missing group header will result in no groups being extracted from the incoming request and a log entry but processing will continue.|null - which means that there will be no group principals extracted from the request and added to the established Subject.
+
+##### Configuration for SiteMinder
+The following is an example of a configuration of the preauthenticated sso provider that leverages the default SM_USER header name - assuming use with CA SiteMinder. It further configures the validation based on the IP address from the incoming request.
+
+	<provider>
+      <role>federation</role>
+      <name>HeaderPreAuth</name>
+      <enabled>true</enabled>
+      <param><name>preauth.validation.method</name><value>preauth.ip.validation</value></param>
+      <param><name>preauth.ip.addresses</name><value>127.0.0.2,127.0.0.1</value></param>
+    </provider>
+
+##### REST Invocation for SiteMinder
+The following curl command can be used to request a directory listing from HDFS while passing in the expected header SM_USER.
+
+	curl -k -i --header "SM_USER: guest" -v https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS
+
+Omitting the --header "SM_USER: guest" above will result in a rejected request.
+
+##### Configuration for IBM Tivoli AM
+As an example for configuring the preauthenticated sso provider for another SSO provider, the following illustrates the values used for IBM's Tivoli Access Manager:
+
+	<provider>
+      <role>federation</role>
+      <name>HeaderPreAuth</name>
+      <enabled>true</enabled>
+      <param><name>preauth.custom.header</name><value>iv_user</value></param>
+      <param><name>preauth.custom.group.header</name><value>iv_group</value></param>
+      <param><name>preauth.validation.method</name><value>preauth.ip.validation</value></param>
+      <param><name>preauth.ip.addresses</name><value>127.0.0.2,127.0.0.1</value></param>
+    </provider>
+
+##### REST Invocation for Tivoli AM
+The following curl command can be used to request a directory listing from HDFS while passing in the expected headers of iv_user and iv_group. Note that the iv_group value in this command matches the expected ACL for webhdfs in the above topology file. Changing this from “admin” to “admin2” should result in a 401 unauthorized response.
+
+	curl -k -i --header "iv_user: guest" --header "iv_group: admin" -v https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS
+
+Omitting the --header "iv_user: guest" above will result in a rejected request.

Added: knox/trunk/books/0.5.0/config_sandbox.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_sandbox.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_sandbox.md (added)
+++ knox/trunk/books/0.5.0/config_sandbox.md Wed May 21 21:19:07 2014
@@ -0,0 +1,51 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+## Sandbox Configuration ##
+
+### Sandbox 2.x Configuration ###
+
+TODO
+
+### Sandbox 1.x Configuration ###
+
+TODO - Update this section to use hostmap if that simplifies things.
+
+This version of the Apache Knox Gateway is tested against [Hortonworks Sandbox 1.x][sandbox]
+
+Currently there is an issue with Sandbox that prevents it from being easily used with the gateway.
+In order to correct the issue, you can use the commands below to login to the Sandbox VM and modify the configuration.
+This assumes that the name sandbox is setup to resolve to the Sandbox VM.
+It may be necessary to use the IP address of the Sandbox VM instead.
+*This is frequently but not always `192.168.56.101`.*
+
+    ssh root@sandbox
+    cp /usr/lib/hadoop/conf/hdfs-site.xml /usr/lib/hadoop/conf/hdfs-site.xml.orig
+    sed -e s/localhost/sandbox/ /usr/lib/hadoop/conf/hdfs-site.xml.orig > /usr/lib/hadoop/conf/hdfs-site.xml
+    shutdown -r now
+
+In addition to make it very easy to follow along with the samples for the gateway you can configure your local system to resolve the address of the Sandbox by the names `vm` and `sandbox`.
+The IP address that is shown below should be that of the Sandbox VM as it is known on your system.
+*This will likely, but not always, be `192.168.56.101`.*
+
+On Linux or Macintosh systems add a line like this to the end of the file `/etc/hosts` on your local machine, *not the Sandbox VM*.
+_Note: The character between the 192.168.56.101 and vm below is a *tab* character._
+
+    192.168.56.101	vm sandbox
+
+On Windows systems a similar but different mechanism can be used.  On recent
+versions of windows the file that should be modified is `%systemroot%\system32\drivers\etc\hosts`

Added: knox/trunk/books/0.5.0/config_webappsec_provider.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/config_webappsec_provider.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/config_webappsec_provider.md (added)
+++ knox/trunk/books/0.5.0/config_webappsec_provider.md Wed May 21 21:19:07 2014
@@ -0,0 +1,66 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Web App Security Provider ###
+Knox is a Web API (REST) Gateway for Hadoop. The fact that REST interactions are HTTP based means that they are vulnerable to a number of web application security vulnerabilities. This project introduces a web application security provider for plugging in various protection filters.
+
+The initial vulnerability protection filter will be for Cross Site Request Forgery (CSRF). Others will be added in future releases.
+ 
+Cross site request forgery (CSRF) attacks attempt to force an authenticated user to 
+execute functionality without their knowledge. By presenting them with a link or image that when clicked invokes a request to another site with which the user may have already established an active session.
+
+CSRF is entirely a browser based attack. Some background knowledge of how browsers work enables us to provide a filter that will prevent CSRF attacks. HTTP requests from a web browser performed via form, image, iframe, etc are unable to set custom HTTP headers. The only way to create a HTTP request from a browser with a custom HTTP header is to use a technology such as Javascript XMLHttpRequest or Flash. These technologies can set custom HTTP headers, but have security policies built in to prevent web sites from sending requests to each other 
+unless specifically allowed by policy. 
+
+This means that a website www.bad.com cannot send a request to  http://bank.example.com with the custom header X-XSRF-Header unless they use a technology such as a XMLHttpRequest. That technology  would prevent such a request from being made unless the bank.example.com domain specifically allowed it. This then results in a REST endpoint that can only be called via XMLHttpRequest (or similar technology).
+
+NOTE: by enabling this protection within the gateway, this custom header will be required for *all* clients that interact with it - not just browsers.
+
+
+#### Configuration ####
+##### Overview #####
+As with all providers in the Knox gateway, the web app security provider is configured through provider params. Unlike many other providers, the web app security provider may actually host multiple vulnerability filters. Currently, we only have an implementation for CSRF but others will follow and you may be interested in creating your own.
+
+Because of this one-to-many provider/filter relationship, there is an extra configuration element for this provider per filter. As you can see in the sample below, the actual filter configuration is defined entirely within the params of the WebAppSec provider.
+
+	<provider
+	  <role>webappsec</role>
+	  <name>WebAppSec</name>
+	  <enabled>true</enabled>
+	  <param><name>csrf.enabled</name><value>true</value></param>
+	  <param><name>csrf.customHeader</name><value>X-XSRF-Header</value></param>
+	  <param><name>csrf.methodsToIgnore</name><value>GET,OPTIONS,HEAD</value></param>
+	</provider>
+
+#### Descriptions ####
+The following table describes the configuration options for the web app security provider:
+
+Name | Description | Default
+---------|-----------
+csrf.enabled|This param enables the CSRF protection capabilities|false  
+csrf.customHeader|This is an optional param that indicates the name of the header to be used in order to determine that the request is from a trusted source. It defaults to the header name described by the NSA in its guidelines for dealing with CSRF in REST.|X-XSRF-Header
+csrf.methodsToIgnore|This is also an optional param that enumerates the HTTP methods to allow through without the custom HTTP header. This is useful for allowing things like GET requests from the URL bar of a browser but it assumes that the GET request adheres to REST principals in terms of being idempotent. If this cannot be assumed then it would be wise to not include GET in the list of methods to ignore.|GET,OPTIONS,HEAD
+
+#### REST Invocation
+The following curl command can be used to request a directory listing from HDFS while passing in the expected header X-XSRF-Header.
+
+	curl -k -i --header "X-XSRF-Header: valid" -v -u guest:guest-password https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS
+
+Omitting the --header "X-XSRF-Header: valid" above should result in an HTTP 400 bad_request.
+
+Disabling the provider will then allow a request that is missing the header through. 
+

Added: knox/trunk/books/0.5.0/knox_cli.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/knox_cli.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/knox_cli.md (added)
+++ knox/trunk/books/0.5.0/knox_cli.md Wed May 21 21:19:07 2014
@@ -0,0 +1,70 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+### Knox CLI ###
+The Knox CLI is a command line utility for management of various aspects of the Knox deployment. It is primarily concerned with the management of the security artifacts for the gateway instance and each of the deployed topologies or hadoop clusters that are gated by the Knox Gateway instance.
+
+The various security artifacts are also generated and populated automatically by the Knox Gateway runtime when they are not found at startup. The assumptions made in those cases are appropriate for a test or development gateway instance and assume 'localhost' for hostname specific activities. For production deployments the use of the CLI may aid in managing  some production deployments.
+
+The knoxcli.sh script is located in the {GATEWAY_HOME}/bin directory.
+
+#### Help ####
+##### knoxcli.sh [--help] #####
+prints help for all commands
+
+#### Master secret persistence ####
+##### knoxcli.sh create-master [--help] #####
+Creates and persists an encrypted master secret in a file within {GATEWAY_HOME}/data/security/master. 
+
+NOTE: This command fails when there is an existing master file in the expected location.
+
+#### Alias creation ####
+##### knoxcli.sh create-alias n [--cluster c] [--value v] [--generate] [--help] #####
+Creates a password alias and stores it in a credential store within the {GATEWAY_HOME}/data/security/keystores dir.  
+
+argument | description
+---------|-----------
+--name|name of the alias to create  
+--cluster|name of Hadoop cluster for the cluster specific credential store otherwise assumes that it is for the gateway itself
+--value|parameter for specifying the actual password otherwise prompted<br/>
+--generate|boolean flag to indicate whether the tool should just generate the value. This assumes that --value is not set - will result in error otherwise. User will not be prompted for the value when --generate is set.		
+
+#### Alias deletion ####
+##### knoxcli.sh delete-alias n [--cluster c] [--help] #####
+Deletes a password and alias mapping from a credential store within {GATEWAY_HOME}/data/security/keystores.  
+
+argument | description
+---------|-----------
+--name | name of the alias to delete  
+--cluster | name of Hadoop cluster for the cluster specific credential store otherwise assumes __gateway
+
+#### Alias listing ####
+##### knoxcli.sh list-alias [--cluster c] [--help] #####
+Lists the alias names for the credential store within {GATEWAY_HOME}/data/security/keystores.  
+
+argument | description
+---------|-----------
+--cluster	|	name of Hadoop cluster for the cluster specific credential store otherwise assumes __gateway
+
+#### Self-signed cert creation ####
+##### knoxcli.sh create-cert [--hostname n] [--help] #####
+Creates and stores a self-signed certificate to represent the identity of the gateway instance. This is stored within the {GATEWAY_HOME}/data/security/keystores/gateway.jks keystore.  
+
+argument | description
+:--------|-----------
+--hostname	|	name of the host to be used in the self-signed certificate. This allows multi-host deployments to specify the proper hostnames for hostname verification to succeed on the client side of the SSL connection. The default is “localhost”.
+

Added: knox/trunk/books/0.5.0/likeised
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/likeised?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/likeised (added)
+++ knox/trunk/books/0.5.0/likeised Wed May 21 21:19:07 2014
@@ -0,0 +1,44 @@
+# This sed script must be kept in sync with the table of contents
+
+#wrap the entire page and the banner
+s@<p><br>  <img src="knox-logo.gif"@<div id="page-wrap"><div id="banner"><p><br>  <img src="knox-logo.gif"@
+
+# close the banner and the start the sidebar
+s@<h2><a id="Table+Of+Contents"></a>Table Of Contents</h2>@</div><div id="sidebar">@
+
+#close the sidebar, start main content section and start the first of the chapers
+s@<h2><a id="Introduction@</div><div id="content"><div id="Introduction"><h2><a id="Introduction@
+s@<h2><a id="Quick+Start@</div><div id="Quick+Start"><h2><a id="Quick+Start@
+s@<h2><a id="Apache+Knox+Details@</div><div id="Apache+Knox+Details"><h2><a id="Apache+Knox+Details@
+# subchapters...
+s@<h4><a id="Apache+Knox+Directory+Layout@</div><div id="Apache+Knox+Directory+Layout"><h4><a id="Layout@
+s@<h3><a id="Supported+Services@</div><div id="Supported+Services"><h3><a id="Supported+Services@
+s@<h4><a id="Configure+Sandbox+port+mapping+for+VirtualBox@</div><div id="Configure+Sandbox+port+mapping+for+VirtualBox"><h4><a id="Configure+Sandbox+port+mapping+for+VirtualBox@
+s@<h2><a id="Gateway+Details@</div><div id="Gateway+Details"><h2><a id="Gateway+Details@
+s@<h3><a id="Configuration@</div><div id="Configuration"><h3><a id="Configuration@
+s@<h3><a id="Knox+CLI@</div><div id="Knox+CLI"><h3><a id="Knox+CLI@
+s@<h3><a id="Authentication@</div><div id="Authentication"><h3><a id="Authentication@
+s@<h3><a id="LDAPGroupLookup@</div><div id="LDAPGroupLookup"><h3><a id="LDAPGroupLookup@
+s@<h3><a id="Identity+Assertion@</div><div id="Identity+Assertion"><h3><a id="Identity+Assertion@
+s@<h3><a id="Authorization@</div><div id="Authorization"><h3><a id="Authorization@
+s@<h2><a id="Configuration@</div><div id="Configuration"><h2><a id="Configuration@
+s@<h3><a id="Secure+Clusters@</div><div id="Secure+Clusters"><h3><a id="Secure+Clusters@
+s@<h3><a id="High+Availability@</div><div id="High+Availability"><h3><a id="High+Availability@
+s@<h3><a id="Web+App+Security+Provider@</div><div id="Web+App+Security+Provider"><h3><a id="Web+App+Security+Provider@
+s@<h3><a id="Preauthenticated+SSO+Provider@</div><div id="Preauthenticated+SSO+Provider"><h3><a id="Preauthenticated+SSO+Provider@
+s@<h3><a id="Audit@</div><div id="Audit"><h3><a id="Audit@
+s@<h2><a id="Client+Details@</div><div id="Client+Details"><h2><a id="Client+Details@
+s@<h2><a id="Service+Details@</div><div id="Service+Details"><h2><a id="Service+Details@
+s@<h3><a id="WebHDFS@</div><div id="WebHDFS"><h3><a id="WebHDFS@
+s@<h3><a id="WebHCat@</div><div id="WebHCat"><h3><a id="WebHCat@
+s@<h3><a id="Oozie@</div><div id="Oozie"><h3><a id="Oozie@
+s@<h3><a id="HBase@</div><div id="HBase"><h3><a id="HBase@
+s@<h3><a id="Hive@</div><div id="Hive"><h3><a id="Hive@
+s@<h2><a id="Limitations@</div><div id="Limitations"><h2><a id="Limitations@
+s@<h2><a id="Troubleshooting@</div><div id="Troubleshooting"><h2><a id="Troubleshooting@
+s@<h2><a id="Export+Controls@</div><div id="Export+Controls"><h2><a id="Export+Controls@
+
+# closing the last chapter section, page-wrap and content sections is done outside of this script
+# using cat >> filename
+
+# sed -f likeised knox-incubating-0-4-0.html > knox-incubating-0-4-0-new.html && echo "</div></div></div>" >> knox-incubating-0-4-0-new.html

Added: knox/trunk/books/0.5.0/quick_start.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.5.0/quick_start.md?rev=1596694&view=auto
==============================================================================
--- knox/trunk/books/0.5.0/quick_start.md (added)
+++ knox/trunk/books/0.5.0/quick_start.md Wed May 21 21:19:07 2014
@@ -0,0 +1,219 @@
+<!---
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+--->
+
+## Quick Start ##
+
+Here are the steps to have Apache Knox up and running against a Hadoop Cluster:
+
+1. Verify system requirements
+1. Download a virtual machine (VM) with Hadoop 
+1. Download Apache Knox Gateway
+1. Start the virtual machine with Hadoop
+1. Install Knox
+1. Start the LDAP embedded within Knox
+1. Start the Knox Gateway
+1. Do Hadoop with Knox
+
+
+
+### 1 - Requirements ###
+
+#### Java ####
+
+Java 1.6 or later is required for the Knox Gateway runtime.
+Use the command below to check the version of Java installed on the system where Knox will be running.
+
+    java -version
+
+#### Hadoop ####
+
+Knox supports Hadoop 1.x or 2.x, the quick start instructions assume a Hadoop 2.x virtual machine based environment. 
+
+
+### 2 - Download Hadoop 2.x VM ###
+The quick start provides a link to download Hadoop 2.0 based Hortonworks virtual machine [Sandbox](http://hortonworks.com/products/hdp-2/#install). Please note Knox supports other Hadoop distributions and is configurable against a full blown Hadoop cluster.
+Configuring Knox for Hadoop 1.x/2.x version, or Hadoop deployed in EC2 or a custom Hadoop cluster is documented in advance deployment guide.
+
+
+### 3 - Download Apache Knox Gateway ###
+
+Download one of the distributions below from the [Apache mirrors][mirror].
+
+* Source archive: [knox-incubating-0.4.0-src.zip][src-zip] ([PGP signature][src-pgp], [SHA1 digest][src-sha], [MD5 digest][src-md5])
+* Binary archive: [knox-incubating-0.4.0.zip][bin-zip] ([PGP signature][bin-pgp], [SHA1 digest][bin-sha], [MD5 digest][bin-md5])
+* RPM package: [knox-incubating-0.4.0.rpm][rpm] ([PGP signature][rpm-pgp], [SHA1 digest][rpm-sha], [MD5 digest][rpm-md5])
+
+[src-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip
+[src-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.sha
+[src-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating-src.zip.asc
+[src-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0-src.zip.md5
+[bin-zip]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip
+[bin-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.asc
+[bin-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.sha
+[bin-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.zip.md5
+[rpm]: http://www.apache.org/dyn/closer.cgi/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm
+[rpm-sha]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.sha
+[rpm-pgp]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-0.4.0-incubating.rpm.asc
+[rpm-md5]: http://www.apache.org/dist/incubator/knox/0.4.0-incubating/knox-incubating-0.4.0.rpm.md5
+
+Apache Knox Gateway releases are available under the [Apache License, Version 2.0][asl].
+See the NOTICE file contained in each release artifact for applicable copyright attribution notices.
+
+
+### Verify ###
+
+While recommended, verify is an optional step. You can verify the integrity of any downloaded files using the PGP signatures.
+Please read [Verifying Apache HTTP Server Releases](http://httpd.apache.org/dev/verification.html) for more information on why you should verify our releases.
+
+The PGP signatures can be verified using PGP or GPG.
+First download the KEYS file as well as the .asc signature files for the relevant release packages.
+Make sure you get these files from the main distribution directory linked above, rather than from a mirror.
+Then verify the signatures using one of the methods below.
+
+    % pgpk -a KEYS
+    % pgpv knox-incubating-0.4.0.zip.asc
+
+or
+
+    % pgp -ka KEYS
+    % pgp knox-incubating-0.4.0.zip.asc
+
+or
+
+    % gpg --import KEYS
+    % gpg --verify knox-incubating-0.4.0.zip.asc
+
+### 4 - Start Hadoop virtual machine ###
+
+Start the Hadoop virtual machine.
+
+### 5 - Install Knox ###
+
+The steps required to install the gateway will vary depending upon which distribution format (zip | rpm) was downloaded.
+In either case you will end up with a directory where the gateway is installed.
+This directory will be referred to as your `{GATEWAY_HOME}` throughout this document.
+
+#### ZIP ####
+
+If you downloaded the Zip distribution you can simply extract the contents into a directory.
+The example below provides a command that can be executed to do this.
+Note the `{VERSION}` portion of the command must be replaced with an actual Apache Knox Gateway version number.
+This might be 0.4.0 for example and must patch the value in the file downloaded.
+
+    jar xf knox-incubating-{VERSION}.zip
+
+This will create a directory `knox-incubating-{VERSION}` in your current directory.
+The directory `knox-incubating-{VERSION}` will considered your `{GATEWAY_HOME}`
+
+
+#### RPM ####
+
+If you downloaded the RPM distribution you can install it using normal RPM package tools.
+It is important that the user that will be running the gateway server is used to install.
+This is because several directories are created that are owned by this user.
+These command will install Knox to `/usr/lib/knox` following the pattern of other Hadoop components.
+This directory will be considered your `{GATEWAY_HOME}`.
+
+    sudo yum localinstall knox-incubating-{VERSION}.rpm
+
+or
+
+    sudo rpm -ihv knox-incubating-{VERSION}.rpm
+
+
+### 6 - Start LDAP embedded in Knox ###
+
+Knox comes with an LDAP server for demonstration purposes.
+
+    cd {GATEWAY_HOME}
+    bin/ldap.sh start
+
+
+### 7 - Start Knox  ###
+
+The gateway can be started in one of two ways, as java -jar or with a shell script.
+
+
+###### Starting via script
+
+Run the knoxcli create-master command in order to persist the master secret that is used to protect the key and credential stores for the gateway instance.
+
+###### linux
+    cd {GATEWAY_HOME}
+    bin/knoxcli.sh create-master
+
+The cli will prompt you for the master secret (i.e. password).
+
+The server will discover the persisted master secret during start up and complete the setup process for demo installs. A demo install will consist of a knox gateway instance with an identity certificate for localhost. This will require clients to be on the same machine or to turn off hostname verification. For more involved deployments, See the Knox CLI section of this document for additional commands - incuding the ability to create a self-signed certificate for a specific hostname.
+
+    cd {GATEWAY_HOME}
+    bin/gateway.sh start
+
+When starting the gateway this way the process will be run in the backgroud.
+The log output is written into the directory /var/log/knox.
+In addition a PID (process ID) is written into /var/run/knox.
+
+In order to stop a gateway that was started with the script use this command.
+
+    cd {GATEWAY_HOME}
+    bin/gateway.sh stop
+
+If for some reason the gateway is stopped other than by using the command above you may need to clear the tracking PID.
+
+    cd {GATEWAY_HOME}
+    bin/gateway.sh clean
+
+__NOTE: This command will also clear any log output in /var/log/knox so use this with caution.__
+
+
+### 8 - Do Hadoop with Knox
+
+#### Put a file in HDFS via Knox.
+#### CAT a file in HDFS via Knox.
+#### Invoke the LISTSATUS operation on WebHDFS via the gateway.
+This will return a directory listing of the root (i.e. /) directory of HDFS.
+
+    curl -i -k -u guest:guest-password -X GET \
+        'https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
+
+The results of the above command should result in something to along the lines of the output below.
+The exact information returned is subject to the content within HDFS in your Hadoop cluster.
+Successfully executing this command at a minimum proves that the gateway is properly configured to provide access to WebHDFS.
+It does not necessarily provide that any of the other services are correct configured to be accessible.
+To validate that see the sections for the individual services in #[Service Details].
+
+    HTTP/1.1 200 OK
+    Content-Type: application/json
+    Content-Length: 760
+    Server: Jetty(6.1.26)
+
+    {"FileStatuses":{"FileStatus":[
+    {"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350595859762,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"type":"DIRECTORY"},
+    {"accessTime":0,"blockSize":0,"group":"mapred","length":0,"modificationTime":1350595874024,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"type":"DIRECTORY"},
+    {"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350596040075,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"type":"DIRECTORY"},
+    {"accessTime":0,"blockSize":0,"group":"hdfs","length":0,"modificationTime":1350595857178,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"type":"DIRECTORY"}
+    ]}}
+    
+#### Submit a MR job via Knox.
+
+#### Get status of a MR job via Knox.
+
+#### Cancel a MR job via Knox.
+
+
+### More Examples ###
+



Mime
View raw message