knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kmin...@apache.org
Subject svn commit: r1542103 - in /incubator/knox: site/ site/books/knox-incubating-0-3-0/ trunk/books/0.3.0/
Date Thu, 14 Nov 2013 22:03:41 GMT
Author: kminder
Date: Thu Nov 14 22:03:41 2013
New Revision: 1542103

URL: http://svn.apache.org/r1542103
Log:
Add Limitations section and some troubleshooting stubs.

Added:
    incubator/knox/trunk/books/0.3.0/book_limitations.md
      - copied, changed from r1542068, incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
Modified:
    incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
    incubator/knox/site/index.html
    incubator/knox/site/issue-tracking.html
    incubator/knox/site/license.html
    incubator/knox/site/mail-lists.html
    incubator/knox/site/project-info.html
    incubator/knox/site/team-list.html
    incubator/knox/trunk/books/0.3.0/book.md
    incubator/knox/trunk/books/0.3.0/book_getting-started.md
    incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
    incubator/knox/trunk/books/0.3.0/quick_start.md

Modified: incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html (original)
+++ incubator/knox/site/books/knox-incubating-0-3-0/knox-incubating-0-3-0.html Thu Nov 14
22:03:41 2013
@@ -16,7 +16,7 @@
 --><p><link href="book.css" rel="stylesheet"/></p>
 <div id="logo" style="width:100%; text-align:center">
   <!--img src="knox-logo.gif" alt="Knox"/-->
-</div><p><br>  <img src="knox-logo.gif" alt="Knox"/>  <img src="apache-incubator-logo.png"
align="right" alt="Incubator"/></p><h1><a id="Apache+Knox+Gateway+0.3.0+(Incubator)+User's+Guide"></a>Apache
Knox Gateway 0.3.0 (Incubator) User&rsquo;s Guide</h1><h2><a id="Table+Of+Contents"></a>Table
Of Contents</h2>
+</div><p><br>  <img src="knox-logo.gif" alt="Knox"/>  <img src="apache-incubator-logo.png"
align="right" alt="Incubator"/></p><h1><a id="Apache+Knox+Gateway+0.3.x+(Incubator)+User's+Guide"></a>Apache
Knox Gateway 0.3.x (Incubator) User&rsquo;s Guide</h1><h2><a id="Table+Of+Contents"></a>Table
Of Contents</h2>
 <ul>
   <li><a href="#Introduction">Introduction</a></li>
   <li><a href="#Quick+Start">Quick Start</a></li>
@@ -44,6 +44,7 @@
     <li><a href="#HBase">HBase</a></li>
     <li><a href="#Hive">Hive</a></li>
   </ul></li>
+  <li><a href="#Limitations">Limitations</a></li>
   <li><a href="#Troubleshooting">Troubleshooting</a></li>
   <li><a href="#Export+Controls">Export Controls</a></li>
 </ul><h2><a id="Introduction"></a>Introduction</h2><p>The
Apache Knox Gateway is a system that provides a single point of authentication and access
for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both
users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control
access and manage the cluster). The gateway runs as a server (or cluster of servers) that
provide centralized access to one or more Hadoop clusters. In general the goals of the gateway
are as follows:</p>
@@ -92,17 +93,23 @@
 </code></pre><p>or</p>
 <pre><code>sudo rpm -ihv knox-incubating-{VERSION}.rpm
 </code></pre><h3><a id="6+-+Start+LDAP+embedded+in+Knox"></a>6
- Start LDAP embedded in Knox</h3><p>Knox comes with an LDAP server for demonstration
purposes.</p>
-<pre><code>java -jar {GATEWAY_HOME}/bin/ldap.jar conf &amp;
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/ldap.jar conf &amp;
 </code></pre><h3><a id="7+-+Start+Knox"></a>7 - Start Knox</h3><p>The
gateway can be started in one of two ways, as java -jar or with a shell script.</p><h6><a
id="Starting+via+Java"></a>Starting via Java</h6><p>This is the simplest
way to start the gateway. Starting this way will result in all logging being written directly
to standard output.</p>
-<pre><code>java -jar {GATEWAY_HOME}/bin/gateway.jar
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/gateway.jar
 </code></pre><p>Upon start, Knox server will prompt you for the master
secret (i.e. password). This secret is used to secure artifacts used by the gateway server
for things like SSL and credential/password aliasing. This secret will have to be entered
at startup unless you choose to persist it.</p><h6><a id="Starting+via+script+(*nix+only)"></a>Starting
via script (*nix only)</h6><p>Run the setup command with root privileges.</p>
-<pre><code>sudo {GATEWAY_HOME}/bin/gateway.sh setup
+<pre><code>cd {GATEWAY_HOME}
+sudo bin/gateway.sh setup
 </code></pre><p>The server will prompt you for the master secret (i.e.
password).</p><p>The server can then be started without root privileges using
this command.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh start
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh start
 </code></pre><p>When starting the gateway this way the process will be
run in the backgroud. The log output is written into the directory /var/log/knox. In addition
a PID (process ID) is written into /var/run/knox.</p><p>In order to stop a gateway
that was started with the script use this command.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh stop
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh stop
 </code></pre><p>If for some reason the gateway is stopped other than by
using the command above you may need to clear the tracking PID.</p>
-<pre><code>{GATEWAY_HOME}/bin/gateway.sh clean
+<pre><code>cd {GATEWAY_HOME}
+bin/gateway.sh clean
 </code></pre><p><strong>NOTE: This command will also clear any log
output in /var/log/knox so use this with caution.</strong></p><h3><a
id="8+-+Do+Hadoop+with+Knox"></a>8 - Do Hadoop with Knox</h3><h4><a
id="Put+a+file+in+HDFS+via+Knox."></a>Put a file in HDFS via Knox.</h4><h4><a
id="CAT+a+file+in+HDFS+via+Knox."></a>CAT a file in HDFS via Knox.</h4><h4><a
id="Invoke+the+LISTSATUS+operation+on+WebHDFS+via+the+gateway."></a>Invoke the LISTSATUS
operation on WebHDFS via the gateway.</h4><p>This will return a directory listing
of the root (i.e. /) directory of HDFS.</p>
 <pre><code>curl -i -k -u guest:guest-password -X GET \
     &#39;https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS&#39;
@@ -270,7 +277,7 @@ Server: Jetty(6.1.26)
       <td><img src="error.png"  alt="n"/> </td>
     </tr>
   </tbody>
-</table><h3><a id="Sandbox+Configuration"></a>Sandbox Configuration</h3><p>TODO</p><h3><a
id="More+Examples"></a>More Examples</h3><p>These examples provide more
detail about how to access various Apache Hadoop services via the Apache Knox Gateway.</p>
+</table><h3><a id="More+Examples"></a>More Examples</h3><p>These
examples provide more detail about how to access various Apache Hadoop services via the Apache
Knox Gateway.</p>
 <ul>
   <li><a href="#WebHDFS+Examples">WebHDFS Examples</a></li>
   <li><a href="#WebHCat+Examples">WebHCat Examples</a></li>
@@ -2255,15 +2262,28 @@ connection.close();
 2012-02-03 --- 18:35:34 --- SampleClass6 --- [TRACE]
 2012-02-03 --- 18:35:34 --- SampleClass2 --- [DEBUG]
 ...
-</code></pre><h2><a id="Troubleshooting"></a>Troubleshooting</h2><h3><a
id="Connection+Errors"></a>Connection Errors</h3><p>TODO - Explain how
to debug connection errors.</p><h3><a id="Enabling+Logging"></a>Enabling
Logging</h3><p>The <code>log4j.properties</code> files <code>{GATEWAY_HOME}/conf</code>
can be used to change the granularity of the logging done by Knox. The Knox server must be
restarted in order for these changes to take effect. There are various useful loggers pre-populated
but commented out.</p>
+</code></pre><h2><a id="Limitations"></a>Limitations</h2><h3><a
id="Secure+Oozie+POST/PUT+Request+Payload+Size+Restriction"></a>Secure Oozie POST/PUT
Request Payload Size Restriction</h3><p>With one exception there are no know size
limits for requests or responses payloads that pass through the gateway. The exception involves
POST or PUT request payload sizes for Oozie in a Kerberos secured Hadoop cluster. In this
one case there is currently a 4Kb payload size limit for the first request made to the Hadoop
cluster. This is a result of how the gateway negotiates a trust relationship between itself
and the cluster via SPNego. There is an undocumented configuration setting to modify this
limit&rsquo;s value if required. In the future this will be made more easily configuration
and at that time it will be documented.</p><h3><a id="LDAP+Groups+Acquisition"></a>LDAP
Groups Acquisition</h3><p>The LDAP authenticator currently does not &ldquo;out
of the box&rdquo; support the acquisition of 
 group information. This can be addressed by implementing a custom Shiro Realm extension.
Building this into the default implementation is on the roadmap.</p><h3><a
id="Group+Membership+Propagation"></a>Group Membership Propagation</h3><p>Groups
that are acquired via Identity Assertion Group Principal Mapping are not propigated to the
Hadoop services. Therefore groups used for Service Level Authorization policy may not match
those acquired within the cluster via GroupMappingServiceProvider plugins.</p><h2><a
id="Troubleshooting"></a>Troubleshooting</h2><h3><a id="Finding+Logs"></a>Finding
Logs</h3><p>When things aren&rsquo;t working the first thing you need to do
is examine the diagnostic logs. Depending upon how you are running the gateway these diagnostic
logs will be output to different locations.</p><h4><a id="java+-jar+bin/gateway.jar"></a>java
-jar bin/gateway.jar</h4><p>When the gateway is run this way the diagnostic output
is written directly to the console. If you want to ca
 pture that output you will need to redirect the console output to a file using OS specific
techniques.</p>
+<pre><code>java -jar bin/gateway.jar &gt; gateway.log
+</code></pre><h4><a id="bin/gateway.sh+start"></a>bin/gateway.sh
start</h4><p>When the gateway is run this way the diagnostic output is written
to /var/log/knox/knox.out and /var/log/knox/knox.err. Typically only knox.out will have content.</p><h3><a
id="Increasing+Logging"></a>Increasing Logging</h3><p>The <code>log4j.properties</code>
files <code>{GATEWAY_HOME}/conf</code> can be used to change the granularity of
the logging done by Knox. The Knox server must be restarted in order for these changes to
take effect. There are various useful loggers pre-populated but commented out.</p>
 <pre><code>log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to
increase the debugging of Apache Knox itself.
 log4j.logger.org.apache.shiro=DEBUG          # Use this logger to increase the debugging
of Apache Shiro.
 log4j.logger.org.apache.http=DEBUG           # Use this logger to increase the debugging
of Apache HTTP components.
 log4j.logger.org.apache.http.client=DEBUG    # Use this logger to increase the debugging
of Apache HTTP client component.
 log4j.logger.org.apache.http.headers=DEBUG   # Use this logger to increase the debugging
of Apache HTTP header.
 log4j.logger.org.apache.http.wire=DEBUG      # Use this logger to increase the debugging
of Apache HTTP wire traffic.
-</code></pre><h3><a id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs
can be filed using <a href="https://issues.apache.org/jira/browse/KNOX">Jira</a>.
Please include the results of this command below in the Environment section. Also include
the version of Hadoop being used in the same section.</p>
-<pre><code>java -jar bin/gateway.jar -version
+</code></pre><h3><a id="LDAP+Server+Connectivity+Issues"></a>LDAP
Server Connectivity Issues</h3><p>If the gateway cannot contact the configured
LDAP server you will see errors in the gateway diagnostic output.</p>
+<pre><code>TODO:Kevin - What does it look like when the LDAP server isn&#39;t
running.
+</code></pre><p>Resolving this will require ensuring that the LDAP server
is running and that connection information is correct. The LDAP server connection information
is configured in the cluster&rsquo;s topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).</p><h3><a
id="Hadoop+Cluster+Connectivity+Issues"></a>Hadoop Cluster Connectivity Issues</h3><p>If
the gateway cannot contact one of the services in the configured Hadoop cluster you will see
errors in the gateway diagnostic output.</p>
+<pre><code>TODO:Kevin - What does it look like when the Sandbox isn&#39;t
running.
+</code></pre><p>Resolving this will require ensuring that the Hadoop services
are running and that connection information is correct. Basic Hadoop connectivity can be evaluated
using cURL as described elsewhere. Otherwise the Hadoop cluster connection information is
configured in the cluster&rsquo;s topology file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).</p><h3><a
id="Check+Hadoop+Cluster+Access+via+cURL"></a>Check Hadoop Cluster Access via cURL</h3><p>When
you are experiencing connectivity issue it can be helpful to &ldquo;bypass&rdquo;
the gateway and invoke the Hadoop REST APIs directly. This can easily be done using the cURL
command line utility or many other REST/HTTP clients. Exactly how to use cURL depends on the
configuration of your Hadoop cluster. In general however you will use a command line the one
that follows.</p>
+<pre><code>curl -ikv -X GET &#39;http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS&#39;
+</code></pre><p>If you are using Sandbox the WebHDFS or NameNode port will
be mapped to localhost so this command can be used.</p>
+<pre><code>curl -ikv -X GET &#39;http://localhost:50070/webhdfs/v1/?op=LISTSTATUS&#39;
+</code></pre><p>If you are using a cluster secured with Kerberos you will
need to have used <code>kinit</code> to authenticate to the KDC. Then the command
below should verify that WebHDFS in the Hadoop cluster is accessible.</p>
+<pre><code>curl -ikv --negotiate -u : -X &#39;http://localhost:50070/webhdfs/v1/?op=LISTSTATUS&#39;
+</code></pre><h3><a id="Authentication+Issues"></a>Authentication
Issues</h3><p>TODO:Kevin - What does it look like when the username/password don&rsquo;t
match what is in LDAP?</p><h3><a id="Hostname+Resolution+Issues"></a>Hostname
Resolution Issues</h3><p>TODO:Kevin - What does it look like when host mapping
is enabled and shouldn&rsquo;t be or vice versa.</p><h3><a id="Job+Submission+Issues+-+HDFS+Home+Directories"></a>Job
Submission Issues - HDFS Home Directories</h3><p>TODO:Dilli - What does it look
like if the LDAP authenticated user doesn&rsquo;t have a HDFS home directory and submits
a job.</p><h3><a id="Job+Submission+Issues+-+OS+Accounts"></a>Job
Submission Issues - OS Accounts</h3><p>TODO:Dilli - What does it look like if
the LDAP authenticated user submits a job but doesn&rsquo;t have an OS account.</p><h3><a
id="HBase+Issues"></a>HBase Issues</h3><p>TODO:Kevin - What does it look
like when HBase/Stargate hangs and how do you fix it.</p><h3><a id="SSL+Certificate+Issues"
 ></a>SSL Certificate Issues</h3><p>TODO:Larry - What does it look like
when a client doesn&rsquo;t trust the gateway&rsquo;s SSL identity certificate?</p><h3><a
id="Filing+Bugs"></a>Filing Bugs</h3><p>Bugs can be filed using <a
href="https://issues.apache.org/jira/browse/KNOX">Jira</a>. Please include the results
of this command below in the Environment section. Also include the version of Hadoop being
used in the same section.</p>
+<pre><code>cd {GATEWAY_HOME}
+java -jar bin/gateway.jar -version
 </code></pre><h2><a id="Export+Controls"></a>Export Controls</h2><p>Apache
Knox Gateway includes cryptographic software. The country in which you currently reside may
have restrictions on the import, possession, use, and/or re-export to another country, of
encryption software. BEFORE using any encryption software, please check your country&rsquo;s
laws, regulations and policies concerning the import, possession, or use, and re-export of
encryption software, to see if this is permitted. See <a href="http://www.wassenaar.org">http://www.wassenaar.org</a>
for more information.</p><p>The U.S. Government Department of Commerce, Bureau
of Industry and Security (BIS), has classified this software as Export Commodity Control Number
(ECCN) 5D002.C.1, which includes information security software using or performing cryptographic
functions with asymmetric algorithms. The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception ENC
  Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations,
Section 740.13) for both object code and source code.</p><p>The following provides
more details on the included cryptographic software:</p>
 <ul>
   <li>Apache Knox Gateway uses the ApacheDS which in turn uses Bouncy Castle generic
encryption libraries.</li>

Modified: incubator/knox/site/index.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/index.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/index.html (original)
+++ incubator/knox/site/index.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/site/issue-tracking.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/issue-tracking.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/issue-tracking.html (original)
+++ incubator/knox/site/issue-tracking.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/site/license.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/license.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/license.html (original)
+++ incubator/knox/site/license.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/site/mail-lists.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/mail-lists.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/mail-lists.html (original)
+++ incubator/knox/site/mail-lists.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/site/project-info.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/project-info.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/project-info.html (original)
+++ incubator/knox/site/project-info.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/site/team-list.html
URL: http://svn.apache.org/viewvc/incubator/knox/site/team-list.html?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/site/team-list.html (original)
+++ incubator/knox/site/team-list.html Thu Nov 14 22:03:41 2013
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 4, 2013 -->
+<!-- Generated by Apache Maven Doxia Site Renderer 1.3 at Nov 14, 2013 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -10,7 +10,7 @@
       @import url("./css/site.css");
     </style>
     <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
-    <meta name="Date-Revision-yyyymmdd" content="20131104" />
+    <meta name="Date-Revision-yyyymmdd" content="20131114" />
     <meta http-equiv="Content-Language" content="en" />
                                                     
 <script type="text/javascript">var _gaq = _gaq || [];
@@ -57,7 +57,7 @@
                         <a href="https://cwiki.apache.org/confluence/display/KNOX/Index"
class="externalLink" title="Wiki">Wiki</a>
               
                     
-                &nbsp;| <span id="publishDate">Last Published: 2013-11-04</span>
+                &nbsp;| <span id="publishDate">Last Published: 2013-11-14</span>
               &nbsp;| <span id="projectVersion">Version: 0.0.0-SNAPSHOT</span>
             </div>
       <div class="clear">

Modified: incubator/knox/trunk/books/0.3.0/book.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book.md (original)
+++ incubator/knox/trunk/books/0.3.0/book.md Thu Nov 14 22:03:41 2013
@@ -24,7 +24,7 @@
   <img src="knox-logo.gif" alt="Knox"/>
   <img src="apache-incubator-logo.png" align="right" alt="Incubator"/>
 
-# Apache Knox Gateway 0.3.0 (Incubator) User's Guide #
+# Apache Knox Gateway 0.3.x (Incubator) User's Guide #
 
 ## Table Of Contents ##
 
@@ -48,6 +48,7 @@
     * #[Oozie]
     * #[HBase]
     * #[Hive]
+* #[Limitations]
 * #[Troubleshooting]
 * #[Export Controls]
 
@@ -72,6 +73,7 @@ In general the goals of the gateway are 
 <<book_gateway-details.md>>
 <<book_client-details.md>>
 <<book_service-details.md>>
+<<book_limitations.md>>
 <<book_troubleshooting.md>>
 
 

Modified: incubator/knox/trunk/books/0.3.0/book_getting-started.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_getting-started.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_getting-started.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_getting-started.md Thu Nov 14 22:03:41 2013
@@ -93,10 +93,6 @@ Only more recent versions of some Hadoop
 |                    | 0.12.0     | ![n]        | ![n]   |
 
 
-### Sandbox Configuration ###
-
-TODO
-
 ### More Examples ###
 
 These examples provide more detail about how to access various Apache Hadoop services via
the Apache Knox Gateway.

Copied: incubator/knox/trunk/books/0.3.0/book_limitations.md (from r1542068, incubator/knox/trunk/books/0.3.0/book_troubleshooting.md)
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_limitations.md?p2=incubator/knox/trunk/books/0.3.0/book_limitations.md&p1=incubator/knox/trunk/books/0.3.0/book_troubleshooting.md&r1=1542068&r2=1542103&rev=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_troubleshooting.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_limitations.md Thu Nov 14 22:03:41 2013
@@ -15,32 +15,28 @@
    limitations under the License.
 --->
 
-## Troubleshooting ##
+## Limitations ##
 
-### Connection Errors ###
 
-TODO - Explain how to debug connection errors.
+### Secure Oozie POST/PUT Request Payload Size Restriction ###
 
+With one exception there are no know size limits for requests or responses payloads that
pass through the gateway.
+The exception involves POST or PUT request payload sizes for Oozie in a Kerberos secured
Hadoop cluster.
+In this one case there is currently a 4Kb payload size limit for the first request made to
the Hadoop cluster.
+This is a result of how the gateway negotiates a trust relationship between itself and the
cluster via SPNego.
+There is an undocumented configuration setting to modify this limit's value if required.
+In the future this will be made more easily configuration and at that time it will be documented.
 
-### Enabling Logging ###
 
-The `log4j.properties` files `{GATEWAY_HOME}/conf` can be used to change the granularity
of the logging done by Knox.
-The Knox server must be restarted in order for these changes to take effect.
-There are various useful loggers pre-populated but commented out.
+### LDAP Groups Acquisition ###
 
-    log4j.logger.org.apache.hadoop.gateway=DEBUG # Use this logger to increase the debugging
of Apache Knox itself.
-    log4j.logger.org.apache.shiro=DEBUG          # Use this logger to increase the debugging
of Apache Shiro.
-    log4j.logger.org.apache.http=DEBUG           # Use this logger to increase the debugging
of Apache HTTP components.
-    log4j.logger.org.apache.http.client=DEBUG    # Use this logger to increase the debugging
of Apache HTTP client component.
-    log4j.logger.org.apache.http.headers=DEBUG   # Use this logger to increase the debugging
of Apache HTTP header.
-    log4j.logger.org.apache.http.wire=DEBUG      # Use this logger to increase the debugging
of Apache HTTP wire traffic.
+The LDAP authenticator currently does not "out of the box" support the acquisition of group
information.
+This can be addressed by implementing a custom Shiro Realm extension.
+Building this into the default implementation is on the roadmap.
 
 
-### Filing Bugs ###
+### Group Membership Propagation ###
 
-Bugs can be filed using [Jira][jira].
-Please include the results of this command below in the Environment section.
-Also include the version of Hadoop being used in the same section.
-
-    java -jar bin/gateway.jar -version
+Groups that are acquired via Identity Assertion Group Principal Mapping are not propigated
to the Hadoop services.
+Therefore groups used for Service Level Authorization policy may not match those acquired
within the cluster via GroupMappingServiceProvider plugins.
 

Modified: incubator/knox/trunk/books/0.3.0/book_troubleshooting.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/book_troubleshooting.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/book_troubleshooting.md (original)
+++ incubator/knox/trunk/books/0.3.0/book_troubleshooting.md Thu Nov 14 22:03:41 2013
@@ -17,12 +17,25 @@
 
 ## Troubleshooting ##
 
-### Connection Errors ###
+### Finding Logs ###
 
-TODO - Explain how to debug connection errors.
+When things aren't working the first thing you need to do is examine the diagnostic logs.
+Depending upon how you are running the gateway these diagnostic logs will be output to different
locations.
 
+#### java -jar bin/gateway.jar ####
 
-### Enabling Logging ###
+When the gateway is run this way the diagnostic output is written directly to the console.
+If you want to capture that output you will need to redirect the console output to a file
using OS specific techniques.
+
+    java -jar bin/gateway.jar > gateway.log
+
+#### bin/gateway.sh start ####
+
+When the gateway is run this way the diagnostic output is written to /var/log/knox/knox.out
and /var/log/knox/knox.err.
+Typically only knox.out will have content.
+
+
+### Increasing Logging ###
 
 The `log4j.properties` files `{GATEWAY_HOME}/conf` can be used to change the granularity
of the logging done by Knox.
 The Knox server must be restarted in order for these changes to take effect.
@@ -36,11 +49,82 @@ There are various useful loggers pre-pop
     log4j.logger.org.apache.http.wire=DEBUG      # Use this logger to increase the debugging
of Apache HTTP wire traffic.
 
 
+### LDAP Server Connectivity Issues ###
+
+If the gateway cannot contact the configured LDAP server you will see errors in the gateway
diagnostic output.
+
+    TODO:Kevin - What does it look like when the LDAP server isn't running.
+
+Resolving this will require ensuring that the LDAP server is running and that connection
information is correct.
+The LDAP server connection information is configured in the cluster's topology file (e.g.
{GATEWAY_HOME}/deployments/sandbox.xml).
+
+
+### Hadoop Cluster Connectivity Issues ###
+
+If the gateway cannot contact one of the services in the configured Hadoop cluster you will
see errors in the gateway diagnostic output.
+
+    TODO:Kevin - What does it look like when the Sandbox isn't running.
+
+Resolving this will require ensuring that the Hadoop services are running and that connection
information is correct.
+Basic Hadoop connectivity can be evaluated using cURL as described elsewhere.
+Otherwise the Hadoop cluster connection information is configured in the cluster's topology
file (e.g. {GATEWAY_HOME}/deployments/sandbox.xml).
+
+
+### Check Hadoop Cluster Access via cURL ###
+
+When you are experiencing connectivity issue it can be helpful to "bypass" the gateway and
invoke the Hadoop REST APIs directly.
+This can easily be done using the cURL command line utility or many other REST/HTTP clients.
+Exactly how to use cURL depends on the configuration of your Hadoop cluster.
+In general however you will use a command line the one that follows.
+
+    curl -ikv -X GET 'http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS'
+
+If you are using Sandbox the WebHDFS or NameNode port will be mapped to localhost so this
command can be used.
+
+    curl -ikv -X GET 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+
+If you are using a cluster secured with Kerberos you will need to have used `kinit` to authenticate
to the KDC.
+Then the command below should verify that WebHDFS in the Hadoop cluster is accessible.
+
+    curl -ikv --negotiate -u : -X 'http://localhost:50070/webhdfs/v1/?op=LISTSTATUS'
+
+
+### Authentication Issues ###
+
+TODO:Kevin - What does it look like when the username/password don't match what is in LDAP?
+
+
+### Hostname Resolution Issues ###
+
+TODO:Kevin - What does it look like when host mapping is enabled and shouldn't be or vice
versa.
+
+
+### Job Submission Issues - HDFS Home Directories ###
+
+TODO:Dilli - What does it look like if the LDAP authenticated user doesn't have a HDFS home
directory and submits a job.
+
+
+### Job Submission Issues - OS Accounts ###
+
+TODO:Dilli - What does it look like if the LDAP authenticated user submits a job but doesn't
have an OS account.
+
+
+### HBase Issues ###
+
+TODO:Kevin - What does it look like when HBase/Stargate hangs and how do you fix it.
+
+
+### SSL Certificate Issues ###
+
+TODO:Larry - What does it look like when a client doesn't trust the gateway's SSL identity
certificate?
+
+
 ### Filing Bugs ###
 
 Bugs can be filed using [Jira][jira].
 Please include the results of this command below in the Environment section.
 Also include the version of Hadoop being used in the same section.
 
+    cd {GATEWAY_HOME}
     java -jar bin/gateway.jar -version
 

Modified: incubator/knox/trunk/books/0.3.0/quick_start.md
URL: http://svn.apache.org/viewvc/incubator/knox/trunk/books/0.3.0/quick_start.md?rev=1542103&r1=1542102&r2=1542103&view=diff
==============================================================================
--- incubator/knox/trunk/books/0.3.0/quick_start.md (original)
+++ incubator/knox/trunk/books/0.3.0/quick_start.md Thu Nov 14 22:03:41 2013
@@ -139,7 +139,8 @@ or
 
 Knox comes with an LDAP server for demonstration purposes.
 
-    java -jar {GATEWAY_HOME}/bin/ldap.jar conf &
+    cd {GATEWAY_HOME}
+    java -jar bin/ldap.jar conf &
 
 
 ### 7 - Start Knox  ###
@@ -152,7 +153,8 @@ The gateway can be started in one of two
 This is the simplest way to start the gateway.
 Starting this way will result in all logging being written directly to standard output.
 
-    java -jar {GATEWAY_HOME}/bin/gateway.jar
+    cd {GATEWAY_HOME}
+    java -jar bin/gateway.jar
 
 
 Upon start, Knox server will prompt you for the master secret (i.e. password).
@@ -164,13 +166,15 @@ This secret will have to be entered at s
 
 Run the setup command with root privileges.
 
-    sudo {GATEWAY_HOME}/bin/gateway.sh setup
+    cd {GATEWAY_HOME}
+    sudo bin/gateway.sh setup
 
 The server will prompt you for the master secret (i.e. password).
 
 The server can then be started without root privileges using this command.
 
-    {GATEWAY_HOME}/bin/gateway.sh start
+    cd {GATEWAY_HOME}
+    bin/gateway.sh start
 
 When starting the gateway this way the process will be run in the backgroud.
 The log output is written into the directory /var/log/knox.
@@ -178,11 +182,13 @@ In addition a PID (process ID) is writte
 
 In order to stop a gateway that was started with the script use this command.
 
-    {GATEWAY_HOME}/bin/gateway.sh stop
+    cd {GATEWAY_HOME}
+    bin/gateway.sh stop
 
 If for some reason the gateway is stopped other than by using the command above you may need
to clear the tracking PID.
 
-    {GATEWAY_HOME}/bin/gateway.sh clean
+    cd {GATEWAY_HOME}
+    bin/gateway.sh clean
 
 __NOTE: This command will also clear any log output in /var/log/knox so use this with caution.__
 



Mime
View raw message