knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pzamp...@apache.org
Subject svn commit: r1825906 - in /knox: site/books/knox-1-1-0/dev-guide.html site/index.html site/issue-tracking.html site/license.html site/mail-lists.html site/project-info.html site/team-list.html trunk/books/1.1.0/dev-guide/book.md
Date Mon, 05 Mar 2018 15:07:49 GMT
Author: pzampino
Date: Mon Mar  5 15:07:49 2018
New Revision: 1825906

URL: http://svn.apache.org/viewvc?rev=1825906&view=rev
Log:
Documented adding service discovery support for Hadoop services

Modified:
    knox/site/books/knox-1-1-0/dev-guide.html
    knox/site/index.html
    knox/site/issue-tracking.html
    knox/site/license.html
    knox/site/mail-lists.html
    knox/site/project-info.html
    knox/site/team-list.html
    knox/trunk/books/1.1.0/dev-guide/book.md

Modified: knox/site/books/knox-1-1-0/dev-guide.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-1-1-0/dev-guide.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/books/knox-1-1-0/dev-guide.html (original)
+++ knox/site/books/knox-1-1-0/dev-guide.html Mon Mar  5 15:07:49 2018
@@ -15,17 +15,28 @@
    limitations under the License.
 --><p><link href="book.css" rel="stylesheet"/></p><p><img src="knox-logo.gif"
alt="Knox"/> <img src="apache-logo.gif" align="right" alt="Apache"/></p><h1><a
id="Apache+Knox+Gateway+1.1.x+Developer's+Guide">Apache Knox Gateway 1.1.x Developer&rsquo;s
Guide</a> <a href="#Apache+Knox+Gateway+1.1.x+Developer's+Guide"><img src="markbook-section-link.png"/></a></h1><h2><a
id="Table+Of+Contents">Table Of Contents</a> <a href="#Table+Of+Contents"><img
src="markbook-section-link.png"/></a></h2>
 <ul>
-  <li><a href="#Overview">Overview</a></li>
-  <li><a href="#Architecture+Overview">Architecture Overview</a></li>
-  <li><a href="#Project+Overview">Project Overview</a></li>
-  <li><a href="#Behavior">Behavior</a></li>
-  <li><a href="#Runtime+Behavior">Runtime Behavior</a></li>
-  <li><a href="#Deployment+Behavior">Deployment Behavior</a></li>
-  <li><a href="#Extension+Points">Extension Points</a></li>
-  <li><a href="#Providers">Providers</a></li>
-  <li><a href="#Services">Services</a></li>
-  <li><a href="#Standard+Providers">Standard Providers</a></li>
-  <li><a href="#Rewrite+Provider">Rewrite Provider</a></li>
+  <li><a href="#Overview">Overview</a>
+  <ul>
+    <li><a href="#Architecture+Overview">Architecture Overview</a></li>
+    <li><a href="#Project+Overview">Project Overview</a></li>
+  </ul></li>
+  <li><a href="#Behavior">Behavior</a>
+  <ul>
+    <li><a href="#Runtime+Behavior">Runtime Behavior</a></li>
+    <li><a href="#Deployment+Behavior">Deployment Behavior</a></li>
+  </ul></li>
+  <li><a href="#Extension+Logistics">Extension Logistics</a>
+  <ul>
+    <li><a href="#Providers">Providers</a></li>
+    <li><a href="#Services">Services</a>
+    <ul>
+      <li><a href="#Service+Discovery">Service Discovery</a></li>
+    </ul></li>
+  </ul></li>
+  <li><a href="#Standard+Providers">Standard Providers</a>
+  <ul>
+    <li><a href="#Rewrite+Provider">Rewrite Provider</a></li>
+  </ul></li>
   <li><a href="#Gateway+Services">Gateway Services</a></li>
   <li><a href="#KnoxSSO+Integration">KnoxSSO Integration</a></li>
   <li><a href="#Health+Monitoring+API">Health Monitoring API</a></li>
@@ -620,7 +631,124 @@ rewrite.xml
    protected void setReplayBufferSize(@Default(&quot;8&quot;) int size) {
       replayBufferSize = size;
    }
-</code></pre><h3><a id="Validator">Validator</a> <a href="#Validator"><img
src="markbook-section-link.png"/></a></h3><p>Apache Knox provides preauth
federation authentication where<br/>Knox supports two built-in validators for verifying
incoming requests. In this section, we describe how to write a custom validator for this scenario.
The provided validators include: </p>
+</code></pre><h3><a id="Service+Discovery">Service Discovery</a>
<a href="#Service+Discovery"><img src="markbook-section-link.png"/></a></h3><p>Knox
supports the ability to dynamically determine endpoint URLs in topologies for supported Hadoop
services. <em>This functionality is currently only supported for Ambari-managed Hadoop
clusters.</em> There are a number of these services, which are officially supported,
but this set can be extended by modifying the source or speciyfing external configuration.</p><h4><a
id="Service+URL+Definitions">Service URL Definitions</a> <a href="#Service+URL+Definitions"><img
src="markbook-section-link.png"/></a></h4><p>The service discovery system
determines service URLs by processing mappings of Hadoop service configuration properties
and corresponding URL templates. The knowledge about converting these arbitrary service configuration
properties into correct service endpoint URLs is defined in a configuration file internal
to the Ambari service disco
 very module.</p><h5><a id="Configuration+Details">Configuration Details</a>
<a href="#Configuration+Details"><img src="markbook-section-link.png"/></a></h5><p>This
internal configuration file ( <strong>ambari-service-discovery-url-mappings.xml</strong>
) in the <strong>gateway-discovery-ambari</strong> module is used to specify a
URL template and the associated configuration properties to populate it. A limited degree
of conditional logic is supported to accommodate things like <em>http</em> vs
<em>https</em> configurations.</p><p>The simplest way to describe
its contents will be by examples.</p><p><strong>Example 1</strong></p><p>The
simplest example of one such mapping involves a service component for which there is a single
configuration property which specifies the complete URL</p>
+<pre><code>&lt;!-- This is the service mapping declaration. The name must
match what is specified as a Knox topology service role --&gt;
+&lt;service name=&quot;OOZIE&quot;&gt;
+
+  &lt;!-- This is the URL pattern with palceholder(s) for values provided by properties
--&gt;
+  &lt;url-pattern&gt;{OOZIE_URL}&lt;/url-pattern&gt;
+
+  &lt;properties&gt;
+
+    &lt;!-- This is a property, which in this simple case, matches a template placeholder
--&gt;
+    &lt;property name=&quot;OOZIE_URL&quot;&gt;
+
+      &lt;!-- This is the component whose configuration will be used to lookup the value
of the subsequent config property name --&gt;
+      &lt;component&gt;OOZIE_SERVER&lt;/component&gt;
+
+      &lt;!-- This is the name of the component config property whose value should be
assigned to the OOZIE_URL value --&gt;
+      &lt;config-property&gt;oozie.base.url&lt;/config-property&gt;
+
+    &lt;/property&gt;
+  &lt;/properties&gt;
+&lt;/service&gt;
+</code></pre><p>The <em>OOZIE_SERVER</em> component configuration
is <strong>oozie-site</strong> If <strong>oozie-site.xml</strong>
has the property named <strong>oozie.base.url</strong> with the value <a href="http://ooziehost:11000">http://ooziehost:11000</a>,
then the resulting URL for the <em>OOZIE</em> service will be <a href="http://ooziehost:11000">http://ooziehost:11000</a></p><p><strong>Example
2</strong></p><p>A slightly more complicated example involves a service
component for which the complete URL is not described by a single detail, but rather multiple
endpoint URL details</p>
+<pre><code>&lt;service name=&quot;WEBHCAT&quot;&gt;
+  &lt;url-pattern&gt;http://{HOST}:{PORT}/templeton&lt;/url-pattern&gt;
+  &lt;properties&gt;
+    &lt;property name=&quot;HOST&quot;&gt;
+      &lt;component&gt;WEBHCAT_SERVER&lt;/component&gt; 
+
+      &lt;!-- This tells discovery to get the hostname for the WEBHCAT_SERVER component
from Ambari --&gt;
+      &lt;hostname/&gt;
+
+    &lt;/property&gt;
+    &lt;property name=&quot;PORT&quot;&gt;
+      &lt;component&gt;WEBHCAT_SERVER&lt;/component&gt;
+
+      &lt;!-- This is the name of the component config property whose value should be
assigned to --&gt;
+      &lt;!-- the PORT value --&gt;
+      &lt;config-property&gt;templeton.port&lt;/config-property&gt;
+
+    &lt;/property&gt;
+  &lt;/properties&gt;
+&lt;/service&gt;
+</code></pre><p><strong>Example 3</strong></p><p>An
even more complicated example involves a service for which <em>HTTPS</em> is supported,
and which employs the limited conditional logic support</p>
+<pre><code>&lt;service name=&quot;ATLAS&quot;&gt;
+  &lt;url-pattern&gt;{SCHEME}://{HOST}:{PORT}&lt;/url-pattern&gt;
+  &lt;properties&gt;
+
+    &lt;!-- Property for getting the ATLAS_SERVER component hostname from Ambari --&gt;
+    &lt;property name=&quot;HOST&quot;&gt;
+      &lt;component&gt;ATLAS_SERVER&lt;/component&gt;
+      &lt;hostname/&gt;
+    &lt;/property&gt;
+
+    &lt;!-- Property for capturing whether TLS is enabled or not; This is not a template
placeholder property --&gt;
+    &lt;property name=&quot;TLS_ENABLED&quot;&gt;
+      &lt;component&gt;ATLAS_SERVER&lt;/component&gt;
+      &lt;config-property&gt;atlas.enableTLS&lt;/config-property&gt;
+    &lt;/property&gt;
+    &lt;!-- Property for getting the http port ; also NOT a template placeholder property
--&gt;
+    &lt;property name=&quot;HTTP_PORT&quot;&gt;
+      &lt;component&gt;ATLAS_SERVER&lt;/component&gt;
+      &lt;config-property&gt;atlas.server.http.port&lt;/config-property&gt;
+    &lt;/property&gt;
+
+    &lt;!-- Property for getting the https port ; also NOT a template placeholder property
--&gt;
+    &lt;property name=&quot;HTTPS_PORT&quot;&gt;
+      &lt;component&gt;ATLAS_SERVER&lt;/component&gt;
+      &lt;config-property&gt;atlas.server.https.port&lt;/config-property&gt;
+    &lt;/property&gt;
+
+    &lt;!-- Template placeholder property, dependent on the TLS_ENABLED property value
--&gt;
+    &lt;property name=&quot;PORT&quot;&gt;
+      &lt;config-property&gt;
+        &lt;if property=&quot;TLS_ENABLED&quot; value=&quot;true&quot;&gt;
+          &lt;then&gt;HTTPS_PORT&lt;/then&gt;
+          &lt;else&gt;HTTP_PORT&lt;/else&gt;
+        &lt;/if&gt;
+      &lt;/config-property&gt;
+    &lt;/property&gt;
+
+    &lt;!-- Template placeholder property, dependent on the TLS_ENABLED property value
--&gt;
+    &lt;property name=&quot;SCHEME&quot;&gt;
+      &lt;config-property&gt;
+        &lt;if property=&quot;TLS_ENABLED&quot; value=&quot;true&quot;&gt;
+          &lt;then&gt;https&lt;/then&gt;
+          &lt;else&gt;http&lt;/else&gt;
+        &lt;/if&gt;
+      &lt;/config-property&gt;
+    &lt;/property&gt;
+  &lt;/properties&gt;
+&lt;/service&gt;
+</code></pre><h5><a id="External+Configuration">External Configuration</a>
<a href="#External+Configuration"><img src="markbook-section-link.png"/></a></h5><p>The
internal configuration for URL construction can be overridden or augmented by way of a configuration
file in the gateway configuration directory, or an alternative file specified by a Java system
property. This mechanism is useful for developing support for new services, for custom solutions,
or any scenario for which rebuilding Knox is not desirable.</p><p>The default
file, for which Knox will search first, is <strong><em>{GATEWAY_HOME}</em>/conf/ambari-discovery-url-mappings.xml</strong></p><p>If
Knox doesn&rsquo;t find that file, it will check for a Java system property named <strong>org.apache.gateway.topology.discovery.ambari.config</strong>,
whose value is the fully-qualified path to an XML file. This file&rsquo;s contents must
adhere to the format outlined above.</p><p>If this configuration exists, Knox
will apply i
 t as if it were part of the internal configuration.</p><p><strong>Example</strong></p><p>If
Apache SOLR weren&rsquo;t supported, then it could be added by creating the following
definition in <strong><em>{GATEWAY_HOME}</em>/conf/ambari-discovery-url-mappings.xml</strong>
:</p>
+<pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
+&lt;service-discovery-url-mappings&gt;
+  &lt;service name=&quot;SOLR&quot;&gt;
+    &lt;url-pattern&gt;http://{HOST}:{PORT}&lt;/url-pattern&gt;
+    &lt;properties&gt;
+      &lt;property name=&quot;HOST&quot;&gt;
+        &lt;component&gt;INFRA_SOLR&lt;/component&gt;
+        &lt;hostname/&gt;
+      &lt;/property&gt;
+      &lt;property name=&quot;PORT&quot;&gt;
+        &lt;component&gt;INFRA_SOLR&lt;/component&gt;
+        &lt;config-property&gt;infra_solr_port&lt;/config-property&gt;
+      &lt;/property&gt;
+    &lt;/properties&gt;
+  &lt;/service&gt;
+&lt;/service-discovery-url-mappings&gt;
+</code></pre><p><strong><em>N.B. Knox must be restarted for
changes to this external configuration to be applied.</em></strong></p><h4><a
id="Component+Configuration+Mapping">Component Configuration Mapping</a> <a href="#Component+Configuration+Mapping"><img
src="markbook-section-link.png"/></a></h4><p>To support URL construction
from service configuration files, Ambari service discovery requires knowledge of the service
component types and their respective relationships to configuration types. This knowledge
is defined in a configuration file internal to the Ambari service discovery module.</p><h5><a
id="Configuration+Details">Configuration Details</a> <a href="#Configuration+Details"><img
src="markbook-section-link.png"/></a></h5><p>This internal configuration
file ( <strong>ambari-service-discovery-component-config-mapping.properties</strong>
) in the <strong>gateway-discovery-ambari</strong> module is used to define the
mapping of Hadoop service component names to the configurat
 ion type from which Knox will lookup property values.</p><p><strong>Example</strong></p>
+<pre><code>NAMENODE=hdfs-site
+RESOURCEMANAGER=yarn-site
+HISTORYSERVER=mapred-site
+OOZIE_SERVER=oozie-site
+HIVE_SERVER=hive-site
+WEBHCAT_SERVER=webhcat-site
+</code></pre><h5><a id="External+Configuration">External Configuration</a>
<a href="#External+Configuration"><img src="markbook-section-link.png"/></a></h5><p>The
internal configuration for component configuration mappings can be overridden or augmented
by way of a configuration file in the gateway configuration directory, or an alternative file
specified by a Java system property. This mechanism is useful for developing support for new
services, for custom solutions, or any scenario for which rebuilding Knox is not desirable.</p><p>The
default file, for which Knox will search first, is <strong><em>{GATEWAY_HOME}</em>/conf/ambari-discovery-component-config.properties</strong>
If Knox doesn&rsquo;t find that file, it will check for a Java system property named <strong>org.apache.knox.gateway.topology.discovery.ambari.component.mapping</strong>,
whose value is the fully-qualified path to a properties file. This file&rsquo;s contents
must adhere to the format outlined above.</p><p>If t
 his configuration exists, Knox will apply it as if it were part of the internal configuration.</p><p><strong>Example</strong></p><p>Following
the aforementioned SOLR example, Knox needs to know in which configuration file to find the
<em>INFRA_SOLR</em> component configuration property, so the following property
must be defined in <strong><em>{GATEWAY_HOME}</em>/conf/ambari-discovery-component-config.properties</strong>
:</p>
+<pre><code>INFRA_SOLR=infra-solr-env
+</code></pre><p>This tells Knox to look for the <strong>infra_solr_port</strong>
property in the <strong>infra-solr-env</strong> configuration.</p><p><strong><em>N.B.
Knox must be restarted for changes to this external configuration to be applied.</em></strong></p><h3><a
id="Validator">Validator</a> <a href="#Validator"><img src="markbook-section-link.png"/></a></h3><p>Apache
Knox provides preauth federation authentication where<br/>Knox supports two built-in
validators for verifying incoming requests. In this section, we describe how to write a custom
validator for this scenario. The provided validators include: </p>
 <ul>
   <li><em>preauth.default.validation:</em> This default behavior does not
perform any validation check. All requests will pass.</li>
   <li><em>preauth.ip.validation</em> : This validation checks if a request
is originated from an IP address which is configured in Knox service through property <em>preauth.ip.addresses</em>.</li>

Modified: knox/site/index.html
URL: http://svn.apache.org/viewvc/knox/site/index.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/index.html (original)
+++ knox/site/index.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Announcing Apache Knox 1.0.0!</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/knox/site/issue-tracking.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/issue-tracking.html (original)
+++ knox/site/issue-tracking.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Issue Tracking</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/site/license.html
URL: http://svn.apache.org/viewvc/knox/site/license.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/license.html (original)
+++ knox/site/license.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Project License</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/knox/site/mail-lists.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/mail-lists.html (original)
+++ knox/site/mail-lists.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Project Mailing Lists</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/site/project-info.html
URL: http://svn.apache.org/viewvc/knox/site/project-info.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/project-info.html (original)
+++ knox/site/project-info.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Project Information</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/site/team-list.html
URL: http://svn.apache.org/viewvc/knox/site/team-list.html?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/site/team-list.html (original)
+++ knox/site/team-list.html Mon Mar  5 15:07:49 2018
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2018-02-22
+ | Generated by Apache Maven Doxia at 2018-03-05
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20180222" />
+    <meta name="Date-Revision-yyyymmdd" content="20180305" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Knox Gateway &#x2013; Team list</title>
     <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -58,7 +58,7 @@
               
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2018-02-22</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2018-03-05</li>

             
                             </ul>
       </div>

Modified: knox/trunk/books/1.1.0/dev-guide/book.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/1.1.0/dev-guide/book.md?rev=1825906&r1=1825905&r2=1825906&view=diff
==============================================================================
--- knox/trunk/books/1.1.0/dev-guide/book.md (original)
+++ knox/trunk/books/1.1.0/dev-guide/book.md Mon Mar  5 15:07:49 2018
@@ -24,16 +24,17 @@
 
 ## Table Of Contents ##
 * #[Overview]
-  * #[Architecture Overview]
-  * #[Project Overview]
+	* #[Architecture Overview]
+	* #[Project Overview]
 * #[Behavior]
-  * #[Runtime Behavior]
-  * #[Deployment Behavior]
-* #[Extension Points]
-  * #[Providers]
-  * #[Services]
+    * #[Runtime Behavior]
+    * #[Deployment Behavior]
+* #[Extension Logistics]
+    * #[Providers]
+    * #[Services]
+		* #[Service Discovery]
 * #[Standard Providers]
-  * #[Rewrite Provider]
+    * #[Rewrite Provider]
 * #[Gateway Services]
 * #[KnoxSSO Integration]
 * #[Health Monitoring API]
@@ -1006,6 +1007,227 @@ org.apache.knox.gateway.dispatch.Default
    }
 ```
 
+### Service Discovery ###
+
+Knox supports the ability to dynamically determine endpoint URLs in topologies for supported
Hadoop services.
+_This functionality is currently only supported for Ambari-managed Hadoop clusters._
+There are a number of these services, which are officially supported, but this set can be
extended by modifying the
+source or speciyfing external configuration.
+
+
+#### Service URL Definitions ####
+
+The service discovery system determines service URLs by processing mappings of Hadoop service
configuration properties and
+corresponding URL templates. The knowledge about converting these arbitrary service configuration
properties into correct
+service endpoint URLs is defined in a configuration file internal to the Ambari service discovery
module.
+
+
+##### Configuration Details #####
+
+This internal configuration file ( **ambari-service-discovery-url-mappings.xml** ) in the
**gateway-discovery-ambari** module is used to specify a
+URL template and the associated configuration properties to populate it. A limited degree
of conditional logic is supported to accommodate
+things like _http_ vs _https_ configurations.
+
+The simplest way to describe its contents will be by examples.
+
+**Example 1**
+
+The simplest example of one such mapping involves a service component for which there is
a single configuration property which specifies
+the complete URL
+
+    <!-- This is the service mapping declaration. The name must match what is specified
as a Knox topology service role -->
+    <service name="OOZIE">
+	
+      <!-- This is the URL pattern with palceholder(s) for values provided by properties
-->
+      <url-pattern>{OOZIE_URL}</url-pattern>
+	
+      <properties>
+	
+        <!-- This is a property, which in this simple case, matches a template placeholder
-->
+        <property name="OOZIE_URL">
+	
+          <!-- This is the component whose configuration will be used to lookup the value
of the subsequent config property name -->
+          <component>OOZIE_SERVER</component>
+	  
+          <!-- This is the name of the component config property whose value should be
assigned to the OOZIE_URL value -->
+          <config-property>oozie.base.url</config-property>
+	  
+        </property>
+      </properties>
+    </service>
+
+The _OOZIE_SERVER_ component configuration is **oozie-site**
+If **oozie-site.xml** has the property named **oozie.base.url** with the value http://ooziehost:11000,
then the resulting URL for the _OOZIE_
+service will be http://ooziehost:11000
+
+
+**Example 2**
+
+A slightly more complicated example involves a service component for which the complete URL
is not described by a single detail, but
+rather multiple endpoint URL details
+
+    <service name="WEBHCAT">
+      <url-pattern>http://{HOST}:{PORT}/templeton</url-pattern>
+      <properties>
+        <property name="HOST">
+          <component>WEBHCAT_SERVER</component> 
+		  
+          <!-- This tells discovery to get the hostname for the WEBHCAT_SERVER component
from Ambari -->
+          <hostname/>
+		  
+        </property>
+        <property name="PORT">
+          <component>WEBHCAT_SERVER</component>
+		  
+          <!-- This is the name of the component config property whose value should be
assigned to -->
+		  <!-- the PORT value -->
+          <config-property>templeton.port</config-property>
+		  
+        </property>
+      </properties>
+    </service>
+
+
+**Example 3**
+
+An even more complicated example involves a service for which _HTTPS_ is supported, and which
employs the limited conditional
+logic support
+
+    <service name="ATLAS">
+      <url-pattern>{SCHEME}://{HOST}:{PORT}</url-pattern>
+      <properties>
+	   
+        <!-- Property for getting the ATLAS_SERVER component hostname from Ambari -->
+        <property name="HOST">
+          <component>ATLAS_SERVER</component>
+          <hostname/>
+        </property>
+		
+        <!-- Property for capturing whether TLS is enabled or not; This is not a template
placeholder property -->
+        <property name="TLS_ENABLED">
+          <component>ATLAS_SERVER</component>
+          <config-property>atlas.enableTLS</config-property>
+        </property>
+        <!-- Property for getting the http port ; also NOT a template placeholder property
-->
+        <property name="HTTP_PORT">
+          <component>ATLAS_SERVER</component>
+          <config-property>atlas.server.http.port</config-property>
+        </property>
+	
+        <!-- Property for getting the https port ; also NOT a template placeholder property
-->
+        <property name="HTTPS_PORT">
+          <component>ATLAS_SERVER</component>
+          <config-property>atlas.server.https.port</config-property>
+        </property>
+	
+        <!-- Template placeholder property, dependent on the TLS_ENABLED property value
-->
+        <property name="PORT">
+          <config-property>
+            <if property="TLS_ENABLED" value="true">
+              <then>HTTPS_PORT</then>
+              <else>HTTP_PORT</else>
+            </if>
+          </config-property>
+        </property>
+	
+        <!-- Template placeholder property, dependent on the TLS_ENABLED property value
-->
+        <property name="SCHEME">
+          <config-property>
+            <if property="TLS_ENABLED" value="true">
+              <then>https</then>
+              <else>http</else>
+            </if>
+          </config-property>
+        </property>
+      </properties>
+    </service>
+
+
+##### External Configuration #####
+The internal configuration for URL construction can be overridden or augmented by way of
a configuration file in the gateway
+configuration directory, or an alternative file specified by a Java system property. This
mechanism is useful for developing
+support for new services, for custom solutions, or any scenario for which rebuilding Knox
is not desirable.
+
+The default file, for which Knox will search first, is **_{GATEWAY_HOME}_/conf/ambari-discovery-url-mappings.xml**
+
+If Knox doesn't find that file, it will check for a Java system property named **org.apache.gateway.topology.discovery.ambari.config**,
+whose value is the fully-qualified path to an XML file. This file's contents must adhere
to the format outlined above.
+
+If this configuration exists, Knox will apply it as if it were part of the internal configuration.
+
+**Example**
+
+If Apache SOLR weren't supported, then it could be added by creating the following definition
in
+**_{GATEWAY_HOME}_/conf/ambari-discovery-url-mappings.xml** :
+
+    <?xml version="1.0" encoding="utf-8"?>
+    <service-discovery-url-mappings>
+      <service name="SOLR">
+        <url-pattern>http://{HOST}:{PORT}</url-pattern>
+        <properties>
+          <property name="HOST">
+            <component>INFRA_SOLR</component>
+            <hostname/>
+          </property>
+          <property name="PORT">
+            <component>INFRA_SOLR</component>
+            <config-property>infra_solr_port</config-property>
+          </property>
+        </properties>
+      </service>
+    </service-discovery-url-mappings>
+
+
+**_N.B. Knox must be restarted for changes to this external configuration to be applied._**
+
+
+#### Component Configuration Mapping ####
+
+To support URL construction from service configuration files, Ambari service discovery requires
knowledge of the service component types and
+their respective relationships to configuration types. This knowledge is defined in a configuration
file internal to the Ambari service discovery
+module.
+
+##### Configuration Details #####
+
+This internal configuration file ( **ambari-service-discovery-component-config-mapping.properties**
) in the **gateway-discovery-ambari** module is
+used to define the mapping of Hadoop service component names to the configuration type from
which Knox will lookup property values.
+
+**Example**
+
+    NAMENODE=hdfs-site
+    RESOURCEMANAGER=yarn-site
+    HISTORYSERVER=mapred-site
+    OOZIE_SERVER=oozie-site
+    HIVE_SERVER=hive-site
+    WEBHCAT_SERVER=webhcat-site
+
+##### External Configuration #####
+
+The internal configuration for component configuration mappings can be overridden or augmented
by way of a configuration file in the gateway
+configuration directory, or an alternative file specified by a Java system property. This
mechanism is useful for developing support for new services,
+for custom solutions, or any scenario for which rebuilding Knox is not desirable.
+
+The default file, for which Knox will search first, is **_{GATEWAY_HOME}_/conf/ambari-discovery-component-config.properties**
+If Knox doesn't find that file, it will check for a Java system property named **org.apache.knox.gateway.topology.discovery.ambari.component.mapping**,
+whose value is the fully-qualified path to a properties file. This file's contents must adhere
to the format outlined above.
+
+If this configuration exists, Knox will apply it as if it were part of the internal configuration.
+
+**Example**
+
+Following the aforementioned SOLR example, Knox needs to know in which configuration file
to find the _INFRA_SOLR_ component configuration property, so
+the following property must be defined in **_{GATEWAY_HOME}_/conf/ambari-discovery-component-config.properties**
:
+
+    
+    INFRA_SOLR=infra-solr-env
+	
+
+This tells Knox to look for the **infra_solr_port** property in the **infra-solr-env** configuration.
+
+**_N.B. Knox must be restarted for changes to this external configuration to be applied._**
+
+
+
 ### Validator ###
 Apache Knox provides preauth federation authentication where  
 Knox supports two built-in validators for verifying incoming requests. In this section, we
describe how to write a custom validator for this scenario. The provided validators include:




Mime
View raw message