knox-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From su...@apache.org
Subject svn commit: r1718240 - /knox/trunk/books/0.7.0/book_ui_service-details.md
Date Mon, 07 Dec 2015 04:15:08 GMT
Author: sumit
Date: Mon Dec  7 04:15:08 2015
New Revision: 1718240

URL: http://svn.apache.org/viewvc?rev=1718240&view=rev
Log:
KNOX-625 docs for UI service support

Added:
    knox/trunk/books/0.7.0/book_ui_service-details.md
      - copied, changed from r1714654, knox/trunk/books/0.7.0/book_service-details.md

Copied: knox/trunk/books/0.7.0/book_ui_service-details.md (from r1714654, knox/trunk/books/0.7.0/book_service-details.md)
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.7.0/book_ui_service-details.md?p2=knox/trunk/books/0.7.0/book_ui_service-details.md&p1=knox/trunk/books/0.7.0/book_service-details.md&r1=1714654&r2=1718240&rev=1718240&view=diff
==============================================================================
--- knox/trunk/books/0.7.0/book_service-details.md (original)
+++ knox/trunk/books/0.7.0/book_ui_service-details.md Mon Dec  7 04:15:08 2015
@@ -15,283 +15,209 @@
    limitations under the License.
 --->
 
-## Service Details ##
+## UI Service Details ##
 
-In the sections that follow the integrations currently available out of the box with the
gateway will be described.
-In general these sections will include examples that demonstrate how to access each of these
services via the gateway.
-In many cases this will include both the use of [cURL][curl] as a REST API client as well
as the use of the Knox Client DSL.
-You may notice that there are some minor differences between using the REST API of a given
service via the gateway.
-In general this is necessary in order to achieve the goal of leaking internal Hadoop cluster
details to the client.
-
-Keep in mind that the gateway uses a plugin model for supporting Hadoop services.
-Check back with the [Apache Knox][site] site for the latest news on plugin availability.
-You can also create your own custom plugin to extend the capabilities of the gateway.
-
-These are the current Hadoop services with built-in support.
-
-* #[WebHDFS]
-* #[WebHCat]
-* #[Oozie]
-* #[HBase]
-* #[Hive]
-* #[Yarn]
-* #[Storm]
+In the sections that follow the integrations for proxying various UIs currently available
out of the box with the
+gateway will be described. These sections will include examples that demonstrate how to access
each of these services
+via the gateway.
+
+These are the current Hadoop services with built-in support for their UIs.
+
+* #[Name Node UI]
+* #[Job History UI]
+* #[Oozie UI]
+* #[HBase UI]
+* #[Yarn UI]
+* #[Spark UI]
 
 ### Assumptions
 
-This document assumes a few things about your environment in order to simplify the examples.
+This section assumes an environment setup similar to the one in the REST services section
#[Service Details]
 
-* The JVM is executable as simply java.
-* The Apache Knox Gateway is installed and functional.
-* The example commands are executed within the context of the GATEWAY_HOME current directory.
-The GATEWAY_HOME directory is the directory within the Apache Knox Gateway installation that
contains the README file and the bin, conf and deployments directories.
-* The [cURL][curl] command line HTTP client utility is installed and functional.
-* A few examples optionally require the use of commands from a standard Groovy installation.
-These examples are optional but to try them you will need Groovy [installed](http://groovy.codehaus.org/Installing+Groovy).
-* The default configuration for all of the samples is setup for use with Hortonworks' [Sandbox][sandbox]
version 2.
-
-### Customization
-
-Using these samples with other Hadoop installations will require changes to the steps describe
here as well as changes to referenced sample scripts.
-This will also likely require changes to the gateway's default configuration.
-In particular host names, ports user names and password may need to be changes to match your
environment.
-These changes may need to be made to gateway configuration and also the Groovy sample script
files in the distribution.
-All of the values that may need to be customized in the sample scripts can be found together
at the top of each of these files.
-
-### cURL
-
-The cURL HTTP client command line utility is used extensively in the examples for each service.
-In particular this form of the cURL command line is used repeatedly.
-
-    curl -i -k -u guest:guest-password ...
-
-The option -i (aka --include) is used to output HTTP response header information.
-This will be important when the content of the HTTP Location header is required for subsequent
requests.
-
-The option -k (aka --insecure) is used to avoid any issues resulting the use of demonstration
SSL certificates.
-
-The option -u (aka --user) is used to provide the credentials to be used when the client
is challenged by the gateway.
-
-Keep in mind that the samples do not use the cookie features of cURL for the sake of simplicity.
-Therefore each request via cURL will result in an authentication.
-
-<<service_webhdfs.md>>
-<<service_webhcat.md>>
-<<service_oozie.md>>
-<<service_hbase.md>>
-<<service_hive.md>>
-<<service_yarn.md>>
-<<service_storm.md>>
-<<service_default_ha.md>>
-
-
-### Service Test API
-
-The gateway supports a Service Test API that can be used to test Knox's ability to connect
to each of the different Hadoop services via a simeple HTTP GET request. To be able to access
this API one must add the following line into the topology for which you wish to run the service
test.
-
-	<service>
-		<role>SERVICE-TEST</role>
-	</service>
-	
-After adding the above to a topology, you can make a cURL request with the following structure
-
-	curl -i -k "https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test?username=guest&password=guest-password"
-
-An alternate method of providing credentials:
-	
-	curl -i -k -u guest:guest-password https://{gateway-hostname}:{gateway-port}/gateway/path/{topology-name}/service-test
-	
-Below is an example response. The gateway is also capable of returning XML if specified in
the request's "Accept" HTTP header.
-	
-	{
-	    "serviceTestWrapper": {
-		   "Tests": {
-			  "ServiceTest": [
-				 {
-					"serviceName": "WEBHDFS",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/webhdfs/v1/?op=LISTSTATUS",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHCAT",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/status",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHCAT",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHCAT",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hive",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHCAT",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/templeton/v1/version/hadoop",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "OOZIE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/build-version",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "OOZIE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/v1/admin/status",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "OOZIE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/oozie/versions",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHBASE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHBASE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/version/cluster",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHBASE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase/status/cluster",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "WEBHBASE",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/hbase",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "RESOURCEMANAGER",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/info",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "RESOURCEMANAGER",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/metrics",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "RESOURCEMANAGER",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/resourcemanager/v1/{topology-name}/apps",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "FALCON",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/stack",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "FALCON",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/admin/version",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "FALCON",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/serialize",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "FALCON",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/vertices/all",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "FALCON",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/falcon/api/metadata/lineage/edges/all",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "STORM",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/configuration",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "STORM",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/cluster/summary",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "STORM",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/supervisor/summary",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 },
-				 {
-					"serviceName": "STORM",
-					"requestURL": "http://{gateway-host}:{gateway-port}/gateway/{topology-name}/storm/api/v1/topology/summary",
-					"responseContent": "Content-Length:0,Content-Type: application/json;charset=utf-8",
-					"httpCode": 200,
-					"message": "Request sucessful."
-				 }
-			  ]
-		   },
-		   "messages": {
-			  "message": [
-
-			  ]
-		   }
-	    }
-	}
-
-	
-We can see that this service-test makes HTTP requests to each of the services through Knox
using the specified topology. The test will only make calls to those services that have entries
within the topology file.
-
-##### Adding and Changing test URLs
-
-URLs for each service are stored in `{GATEWAY_HOME}/data/services/{service-name}/{service-version}/service.xml`.
Each `<testURL>` element represents a service resource that will be tested if the service
is set up in the topology. You can add or remove these from teh service.xml files. Just note
if you add URL's there is no guarantee in the order they will be tested. All default URLs
have been tested and work on various clusters. If a new URL is added and doesn't respond in
a way the user expects then it is up to the user to determine whether the URL is correct or
not.
-
-##### Some important things to note:
- - In the first cURL request, the quotes are necessary around the URL or else a command line
terminal will not include the `&password` query parameter in the request.
- - This API call does not require any credentials to receive a response from Knox, but expect
to receive 401 responses from each of the services if none are provided.
+### Name Node UI ###
+
+The Name Node UI is available on the same host and port combination that WebHDFS is available
on. As mentioned in the
+WebHDFS REST service configuration section, the values for the host and port can be obtained
from the following
+properties in hdfs-site.xml
+
+    <property>
+        <name>dfs.namenode.http-address</name>
+        <value>sandbox.hortonworks.com:50070</value>
+    </property>
+    <property>
+        <name>dfs.https.namenode.https-address</name>
+        <value>sandbox.hortonworks.com:50470</value>
+    </property>
+
+The values above need to be reflected in each topology descriptor file deployed to the gateway.
+The gateway by default includes a sample topology descriptor file `{GATEWAY_HOME}/deployments/sandbox.xml`.
+The values in this sample are configured to work with an installed Sandbox VM.
+
+    <service>
+        <role>HDFSUI</role>
+        <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+    </service>
+
+In addition to the service configuration for HDFSUI, the REST service configuration for WEBHDFS
is also required.
+
+    <service>
+        <role>NAMENODE</role>
+        <url>hdfs://sandbox.hortonworks.com:8020</url>
+    </service>
+    <service>
+        <role>WEBHDFS</role>
+        <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+    </service>
+
+By default the gateway is configured to use the HTTP endpoint for WebHDFS in the Sandbox.
+This could alternatively be configured to use the HTTPS endpoint by provided the correct
address.
+
+#### Name Node UI URL Mapping ####
+
+For Name Node UI URLs, the mapping of Knox Gateway accessible HDFS UI URLs to direct HDFS
UI URLs is:
+
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hdfs` |
+| Cluster | `http://{webhdfs-host}:50070/`                                             |
+
+For example to browse the file system using the NameNode UI the URL in a web browser would
be:
+
+    http://sandbox.hortonworks.com:50070/explorer.html#
+
+And using the gateway to access the same page the URL would be (where the gateway host:port
is 'localhost:8443')
+
+    https://localhost:8443/gateway/sandbox/hdfs/explorer.html#
+
+
+### Job History UI ###
+
+The Job History UI service can be configured in a topology by adding the following snippet.
The values in this sample
+are configured to work with an installed Sandbox VM.
+
+    <service>
+        <role>JOBHISTORYUI</role>
+        <url>http://sandbox.hortonworks.com:19888</url>
+    </service>
+
+The values for the host and port can be obtained from the following property in mapred-site.xml
+
+    <property>
+        <name>mapreduce.jobhistory.webapp.address</name>
+        <value>sandbox.hortonworks.com:19888</value>
+    </property>
+
+
+
+#### Job History UI URL Mapping ####
+
+For Job History UI URLs, the mapping of Knox Gateway accessible Job History UI URLs to direct
Job History UI URLs is:
+
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/jobhistory`
|
+| Cluster | `http://{jobhistory-host}:19888/jobhistory`                                 
    |
+
+
+### Oozie UI ###
+
+The Oozie UI service can be configured in a topology by adding the following snippet. The
values in this sample
+are configured to work with an installed Sandbox VM.
+
+    <service>
+        <role>OOZIEUI</role>
+        <url>http://sandbox.hortonworks.com:11000/oozie</url>
+    </service>
+
+The value for the URL can be obtained from the following property in oozie-site.xml
+
+    <property>
+        <name>oozie.base.url</name>
+        <value>http://sandbox.hortonworks.com:11000/oozie</value>
+    </property>
+
+
+
+#### Oozie UI URL Mapping ####
+
+For Oozie UI URLs, the mapping of Knox Gateway accessible Oozie UI URLs to direct Oozie UI
URLs is:
+
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/oozie/`
|
+| Cluster | `http://{oozie-host}:11000/oozie/`                                          
  |
+
+
+### HBase UI ###
+
+The HBase UI service can be configured in a topology by adding the following snippet. The
values in this sample
+are configured to work with an installed Sandbox VM.
+
+    <service>
+        <role>HBASEUI</role>
+        <url>http://sandbox.hortonworks.com:16010</url>
+    </service>
+
+The values for the host and port can be obtained from the following property in hbase-site.xml.
+Below the hostname of the hbase master is used since the bindAddress is 0.0.0.0
+
+    <property>
+        <name>hbase.master.info.bindAddress</name>
+        <value>0.0.0.0</value>
+    </property>
+    <property>
+        <name>hbase.master.info.port</name>
+        <value>16010</value>
+    </property>
+
+#### HBase UI URL Mapping ####
+
+For HBase UI URLs, the mapping of Knox Gateway accessible HBase UI URLs to direct HBase Master
+UI URLs is:
+
+| ------- | -------------------------------------------------------------------------------------|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/hbase/webui/`
  |
+| Cluster | `http://{hbase-master-host}:16010/`                                         
        |
+
+### Yarn UI ###
+
+The Yarn UI service can be configured in a topology by adding the following snippet. The
values in this sample
+are configured to work with an installed Sandbox VM.
+
+    <service>
+        <role>YARNUI</role>
+        <url>http://sandbox.hortonworks.com:8088</url>
+    </service>
+
+The values for the host and port can be obtained from the following property in mapred-site.xml
+
+    <property>
+        <name>yarn.resourcemanager.webapp.address</name>
+        <value>sandbox.hortonworks.com:8088</value>
+    </property>
+
+#### Yarn UI URL Mapping ####
+
+For Resource Manager UI URLs, the mapping of Knox Gateway accessible Resource Manager UI
URLs to direct Resource Manager
+UI URLs is:
+
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/yarn`  
 |
+| Cluster | `http://{resource-manager-host}:8088/cluster`                               
 |
+
+### Spark UI ###
+
+The Spark History UI service can be configured in a topology by adding the following snippet.
The values in this sample
+are configured to work with an installed Sandbox VM.
+
+
+    <service>
+        <role>SPARKHISTORYUI</role>
+        <url>http://sandbox.hortonworks.com:18080/</url>
+    </service>
+
+#### Spark History UI URL Mapping ####
+
+For Spark History UI URLs, the mapping of Knox Gateway accessible Spark History UI URLs to
direct Spark History
+UI URLs is:
+
+| ------- | -----------------------------------------------------------------------------
|
+| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/sparkhistory`
   |
+| Cluster | `http://{spark-history-host}:18080`                                         
         |



Mime
View raw message