knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From su...@apache.org
Subject svn commit: r1718414 - in /knox: site/books/knox-0-7-0/user-guide.html trunk/books/0.7.0/book.md trunk/books/0.7.0/book_ui_service-details.md trunk/books/0.7.0/book_ui_service_details.md
Date Mon, 07 Dec 2015 15:55:08 GMT
Author: sumit
Date: Mon Dec  7 15:55:08 2015
New Revision: 1718414

URL: http://svn.apache.org/viewvc?rev=1718414&view=rev
Log:
KNOX-625 Fix page link to ui services

Added:
    knox/trunk/books/0.7.0/book_ui_service_details.md
      - copied unchanged from r1718407, knox/trunk/books/0.7.0/book_ui_service-details.md
Removed:
    knox/trunk/books/0.7.0/book_ui_service-details.md
Modified:
    knox/site/books/knox-0-7-0/user-guide.html
    knox/trunk/books/0.7.0/book.md

Modified: knox/site/books/knox-0-7-0/user-guide.html
URL: http://svn.apache.org/viewvc/knox/site/books/knox-0-7-0/user-guide.html?rev=1718414&r1=1718413&r2=1718414&view=diff
==============================================================================
--- knox/site/books/knox-0-7-0/user-guide.html (original)
+++ knox/site/books/knox-0-7-0/user-guide.html Mon Dec  7 15:55:08 2015
@@ -4196,7 +4196,163 @@ curl -ik -b ~/cookiejar.txt -c ~/cookiej
 <ul>
   <li>In the first cURL request, the quotes are necessary around the URL or else a
command line terminal will not include the <code>&amp;password</code> query
parameter in the request.</li>
   <li>This API call does not require any credentials to receive a response from Knox,
but expect to receive 401 responses from each of the services if none are provided.</li>
-</ul><h2><a id="Limitations">Limitations</a> <a href="#Limitations"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction">Secure
Oozie POST/PUT Request Payload Size Restriction</a> <a href="#Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"><img
src="markbook-section-link.png"/></a></h3><p>With one exception there
are no known size limits for requests or responses payloads that pass through the gateway.
The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop
cluster. In this one case there is currently a 4Kb payload size limit for the first request
made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship
between itself and the cluster via SPNego. There is an undocumented configuration setting
to modify this limit&rsquo;s value if required. In the future this will be made more easily
configuration and at that time it will be documented.</p><
 h3><a id="Group+Membership+Propagation">Group Membership Propagation</a> <a
href="#Group+Membership+Propagation"><img src="markbook-section-link.png"/></a></h3><p>Groups
that are acquired via Shiro Group Lookup and/or Identity Assertion Group Principal Mapping
are not propagated to the Hadoop services. Therefore, groups used for Service Level Authorization
policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting">Troubleshooting</a> <a href="#Troubleshooting"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Finding+Logs">Finding
Logs</a> <a href="#Finding+Logs"><img src="markbook-section-link.png"/></a></h3><p>When
things aren&rsquo;t working the first thing you need to do is examine the diagnostic logs.
Depending upon how you are running the gateway these diagnostic logs will be output to different
locations.</p><h4><a id="java+-jar+bin/gateway.jar">java -jar bin/gateway.jar</a>
<a href="#java+-jar+bin/ga
 teway.jar"><img src="markbook-section-link.png"/></a></h4><p>When
the gateway is run this way the diagnostic output is written directly to the console. If you
want to capture that output you will need to redirect the console output to a file using OS
specific techniques.</p>
+</ul><h2><a id="UI+Service+Details">UI Service Details</a> <a
href="#UI+Service+Details"><img src="markbook-section-link.png"/></a></h2><p>In
the sections that follow the integrations for proxying various UIs currently available out
of the box with the gateway will be described. These sections will include examples that demonstrate
how to access each of these services via the gateway.</p><p>These are the current
Hadoop services with built-in support for their UIs.</p>
+<ul>
+  <li><a href="#Name+Node+UI">Name Node UI</a></li>
+  <li><a href="#Job+History+UI">Job History UI</a></li>
+  <li><a href="#Oozie+UI">Oozie UI</a></li>
+  <li><a href="#HBase+UI">HBase UI</a></li>
+  <li><a href="#Yarn+UI">Yarn UI</a></li>
+  <li><a href="#Spark+UI">Spark UI</a></li>
+</ul><h3><a id="Assumptions">Assumptions</a> <a href="#Assumptions"><img
src="markbook-section-link.png"/></a></h3><p>This section assumes an
environment setup similar to the one in the REST services section <a href="#Service+Details">Service
Details</a></p><h3><a id="Name+Node+UI">Name Node UI</a> <a
href="#Name+Node+UI"><img src="markbook-section-link.png"/></a></h3><p>The
Name Node UI is available on the same host and port combination that WebHDFS is available
on. As mentioned in the WebHDFS REST service configuration section, the values for the host
and port can be obtained from the following properties in hdfs-site.xml</p>
+<pre><code>&lt;property&gt;
+    &lt;name&gt;dfs.namenode.http-address&lt;/name&gt;
+    &lt;value&gt;sandbox.hortonworks.com:50070&lt;/value&gt;
+&lt;/property&gt;
+&lt;property&gt;
+    &lt;name&gt;dfs.https.namenode.https-address&lt;/name&gt;
+    &lt;value&gt;sandbox.hortonworks.com:50470&lt;/value&gt;
+&lt;/property&gt;
+</code></pre><p>The values above need to be reflected in each topology
descriptor file deployed to the gateway. The gateway by default includes a sample topology
descriptor file <code>{GATEWAY_HOME}/deployments/sandbox.xml</code>. The values
in this sample are configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;HDFSUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:50070/webhdfs&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>In addition to the service configuration for HDFSUI, the
REST service configuration for WEBHDFS is also required.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;NAMENODE&lt;/role&gt;
+    &lt;url&gt;hdfs://sandbox.hortonworks.com:8020&lt;/url&gt;
+&lt;/service&gt;
+&lt;service&gt;
+    &lt;role&gt;WEBHDFS&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:50070/webhdfs&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>By default the gateway is configured to use the HTTP endpoint
for WebHDFS in the Sandbox. This could alternatively be configured to use the HTTPS endpoint
by provided the correct address.</p><h4><a id="Name+Node+UI+URL+Mapping">Name
Node UI URL Mapping</a> <a href="#Name+Node+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
Name Node UI URLs, the mapping of Knox Gateway accessible HDFS UI URLs to direct HDFS UI URLs
is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hdfs</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{webhdfs-host}:50070/</code> </td>
+    </tr>
+  </tbody>
+</table><p>For example to browse the file system using the NameNode UI the URL
in a web browser would be:</p>
+<pre><code>http://sandbox.hortonworks.com:50070/explorer.html#
+</code></pre><p>And using the gateway to access the same page the URL would
be (where the gateway host:port is &lsquo;localhost:8443&rsquo;)</p>
+<pre><code>https://localhost:8443/gateway/sandbox/hdfs/explorer.html#
+</code></pre><h3><a id="Job+History+UI">Job History UI</a>
<a href="#Job+History+UI"><img src="markbook-section-link.png"/></a></h3><p>The
Job History UI service can be configured in a topology by adding the following snippet. The
values in this sample are configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;JOBHISTORYUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:19888&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>The values for the host and port can be obtained from the
following property in mapred-site.xml</p>
+<pre><code>&lt;property&gt;
+    &lt;name&gt;mapreduce.jobhistory.webapp.address&lt;/name&gt;
+    &lt;value&gt;sandbox.hortonworks.com:19888&lt;/value&gt;
+&lt;/property&gt;
+</code></pre><h4><a id="Job+History+UI+URL+Mapping">Job History UI
URL Mapping</a> <a href="#Job+History+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
Job History UI URLs, the mapping of Knox Gateway accessible Job History UI URLs to direct
Job History UI URLs is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/jobhistory</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{jobhistory-host}:19888/jobhistory</code> </td>
+    </tr>
+  </tbody>
+</table><h3><a id="Oozie+UI">Oozie UI</a> <a href="#Oozie+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Oozie UI service can
be configured in a topology by adding the following snippet. The values in this sample are
configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;OOZIEUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:11000/oozie&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>The value for the URL can be obtained from the following
property in oozie-site.xml</p>
+<pre><code>&lt;property&gt;
+    &lt;name&gt;oozie.base.url&lt;/name&gt;
+    &lt;value&gt;http://sandbox.hortonworks.com:11000/oozie&lt;/value&gt;
+&lt;/property&gt;
+</code></pre><h4><a id="Oozie+UI+URL+Mapping">Oozie UI URL Mapping</a>
<a href="#Oozie+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
Oozie UI URLs, the mapping of Knox Gateway accessible Oozie UI URLs to direct Oozie UI URLs
is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{oozie-host}:11000/oozie/</code> </td>
+    </tr>
+  </tbody>
+</table><h3><a id="HBase+UI">HBase UI</a> <a href="#HBase+UI"><img
src="markbook-section-link.png"/></a></h3><p>The HBase UI service can
be configured in a topology by adding the following snippet. The values in this sample are
configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;HBASEUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:16010&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>The values for the host and port can be obtained from the
following property in hbase-site.xml. Below the hostname of the hbase master is used since
the bindAddress is 0.0.0.0</p>
+<pre><code>&lt;property&gt;
+    &lt;name&gt;hbase.master.info.bindAddress&lt;/name&gt;
+    &lt;value&gt;0.0.0.0&lt;/value&gt;
+&lt;/property&gt;
+&lt;property&gt;
+    &lt;name&gt;hbase.master.info.port&lt;/name&gt;
+    &lt;value&gt;16010&lt;/value&gt;
+&lt;/property&gt;
+</code></pre><h4><a id="HBase+UI+URL+Mapping">HBase UI URL Mapping</a>
<a href="#HBase+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
HBase UI URLs, the mapping of Knox Gateway accessible HBase UI URLs to direct HBase Master
UI URLs is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase/webui/</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{hbase-master-host}:16010/</code> </td>
+    </tr>
+  </tbody>
+</table><h3><a id="Yarn+UI">Yarn UI</a> <a href="#Yarn+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Yarn UI service can
be configured in a topology by adding the following snippet. The values in this sample are
configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;YARNUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:8088&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><p>The values for the host and port can be obtained from the
following property in mapred-site.xml</p>
+<pre><code>&lt;property&gt;
+    &lt;name&gt;yarn.resourcemanager.webapp.address&lt;/name&gt;
+    &lt;value&gt;sandbox.hortonworks.com:8088&lt;/value&gt;
+&lt;/property&gt;
+</code></pre><h4><a id="Yarn+UI+URL+Mapping">Yarn UI URL Mapping</a>
<a href="#Yarn+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
Resource Manager UI URLs, the mapping of Knox Gateway accessible Resource Manager UI URLs
to direct Resource Manager UI URLs is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/yarn</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{resource-manager-host}:8088/cluster</code> </td>
+    </tr>
+  </tbody>
+</table><h3><a id="Spark+UI">Spark UI</a> <a href="#Spark+UI"><img
src="markbook-section-link.png"/></a></h3><p>The Spark History UI service
can be configured in a topology by adding the following snippet. The values in this sample
are configured to work with an installed Sandbox VM.</p>
+<pre><code>&lt;service&gt;
+    &lt;role&gt;SPARKHISTORYUI&lt;/role&gt;
+    &lt;url&gt;http://sandbox.hortonworks.com:18080/&lt;/url&gt;
+&lt;/service&gt;
+</code></pre><h4><a id="Spark+History+UI+URL+Mapping">Spark History
UI URL Mapping</a> <a href="#Spark+History+UI+URL+Mapping"><img src="markbook-section-link.png"/></a></h4><p>For
Spark History UI URLs, the mapping of Knox Gateway accessible Spark History UI URLs to direct
Spark History UI URLs is:</p>
+<table>
+  <tbody>
+    <tr>
+      <td>Gateway </td>
+      <td><code>https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/sparkhistory</code>
</td>
+    </tr>
+    <tr>
+      <td>Cluster </td>
+      <td><code>http://{spark-history-host}:18080</code> </td>
+    </tr>
+  </tbody>
+</table><h2><a id="Limitations">Limitations</a> <a href="#Limitations"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction">Secure
Oozie POST/PUT Request Payload Size Restriction</a> <a href="#Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"><img
src="markbook-section-link.png"/></a></h3><p>With one exception there
are no known size limits for requests or responses payloads that pass through the gateway.
The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop
cluster. In this one case there is currently a 4Kb payload size limit for the first request
made to the Hadoop cluster. This is a result of how the gateway negotiates a trust relationship
between itself and the cluster via SPNego. There is an undocumented configuration setting
to modify this limit&rsquo;s value if required. In the future this will be made more easily
configuration and at that time it will be documented.</
 p><h3><a id="Group+Membership+Propagation">Group Membership Propagation</a>
<a href="#Group+Membership+Propagation"><img src="markbook-section-link.png"/></a></h3><p>Groups
that are acquired via Shiro Group Lookup and/or Identity Assertion Group Principal Mapping
are not propagated to the Hadoop services. Therefore, groups used for Service Level Authorization
policy may not match those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting">Troubleshooting</a> <a href="#Troubleshooting"><img
src="markbook-section-link.png"/></a></h2><h3><a id="Finding+Logs">Finding
Logs</a> <a href="#Finding+Logs"><img src="markbook-section-link.png"/></a></h3><p>When
things aren&rsquo;t working the first thing you need to do is examine the diagnostic logs.
Depending upon how you are running the gateway these diagnostic logs will be output to different
locations.</p><h4><a id="java+-jar+bin/gateway.jar">java -jar bin/gateway.jar</a>
<a href="#java+-jar+bin
 /gateway.jar"><img src="markbook-section-link.png"/></a></h4><p>When
the gateway is run this way the diagnostic output is written directly to the console. If you
want to capture that output you will need to redirect the console output to a file using OS
specific techniques.</p>
 <pre><code>java -jar bin/gateway.jar &gt; gateway.log
 </code></pre><h4><a id="bin/gateway.sh+start">bin/gateway.sh start</a>
<a href="#bin/gateway.sh+start"><img src="markbook-section-link.png"/></a></h4><p>When
the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out and
/var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a
id="Increasing+Logging">Increasing Logging</a> <a href="#Increasing+Logging"><img
src="markbook-section-link.png"/></a></h3><p>The <code>log4j.properties</code>
files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of
the logging done by Knox. The Knox server must be restarted in order for these changes to
take effect. There are various useful loggers pre-populated but commented out.</p>
 <pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to
increase the debugging of Apache Knox itself.

Modified: knox/trunk/books/0.7.0/book.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book.md?rev=1718414&r1=1718413&r2=1718414&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book.md (original)
+++ knox/trunk/books/0.7.0/book.md Mon Dec  7 15:55:08 2015
@@ -85,6 +85,7 @@ In general the goals of the gateway are
 <<book_gateway-details.md>>
 <<book_client-details.md>>
 <<book_service-details.md>>
+<<book_ui_service_details.md>>
 <<book_limitations.md>>
 <<book_troubleshooting.md>>
 



Mime
View raw message