In the sections that follow, the integrations currently available out of the box with the gateway will be described. In general these sections will include examples that demonstrate how to access each of these services via the gateway. In many cases this will include both the use of cURL as a REST API client as well as the use of the Knox Client DSL. You may notice that there are some minor differences between using the REST API of a given service via the gateway. In general this is necessary in order to achieve the goal of not leaking internal Hadoop cluster details to the client.
Keep in mind that the gateway uses a plugin model for supporting Hadoop services. Check back with the Apache Knox site for the latest news on plugin availability. You can also create your own custom p lugin to extend the capabilities of the gateway.
These are the current Hadoop services with built-in support.
+ +In the sections that follow, the integrations currently available out of the box with the gateway will be described. In general these sections will include examples that demonstrate how to access each of these services via the gateway. In many cases this will include both the use of cURL as a REST API client as well as the use of the Knox Client DSL. You may notice that there are some minor differences between using the REST API of a given service via the gateway. In general this is necessary in order to achieve the goal of not leaking internal Hadoop cluster details to the client.
Keep in mind that the gateway uses a plugin model for supporting Hadoop services. Check back with the Apache Knox site for the latest news on plugin availability. You can also create your own custom plu gin to extend the capabilities of the gateway.
These are the current Hadoop services with built-in support.
Using these samples with other Hadoop installations will require changes to the steps described here as well as changes to referenced sample scripts. This will also likely require changes to the gateway’s default configuration. In particular host names, ports, user names and password may need to be changed to match your environment. These changes may need to be made to gateway configuration and also the Groovy sample script files in the distribution. All of the values that may need to be customized in the sample scripts can be found together at the top of each of these files.
The cURL HTTP client command line utility is used extensively in the examples for each service. In particular this form of the cURL command line is used repeatedly.
curl -i -k -u guest:guest-password ...
-The option -i (aka –include) is used to output HTTP response header information. This will be important when the content of the HTTP Location header is required for subsequent requests.
The option -k (aka –insecure) is used to avoid any issues resulting from the use of demonstration SSL certificates.
The option -u (aka –user) is used to provide the credentials to be used when the client is challenged by the gateway.
Keep in mind that the samples do not use the cookie features of cURL for the sake of simplicity. Therefore each request via cURL will result in an authentication.
REST API access to HDFS in a Hadoop cluster is provided by WebHDFS. The WebHDFS REST API documentation is available online. WebHDFS must be enabled in the hdfs-si
te.xml configuration file. In the sandbox this configuration file is located at /etc/hadoop/conf/hdfs-site.xml. Note the properties shown below as they are related to configuration required by the gateway. Some of these represent the default values and may not actually be present in hdfs-site.xml.
The option -i (aka –include) is used to output HTTP response header information. This will be important when the content of the HTTP Location header is required for subsequent requests.
The option -k (aka –insecure) is used to avoid any issues resulting from the use of demonstration SSL certificates.
The option -u (aka –user) is used to provide the credentials to be used when the client is challenged by the gateway.
Keep in mind that the samples do not use the cookie features of cURL for the sake of simplicity. Therefore each request via cURL will result in an authentication.
+REST API access to HDFS in a Hadoop cluster is provided by WebHDFS. The WebHDFS REST API documentation is available online. WebHDFS must be enabled in the hdfs-site.xml configuration file. In the sandbox this configuration file is located at /etc/hadoop/conf/hdfs-site.xml. Note the properties shown below as they are related to configuration required by the gateway. Some of these represent the default values and may not actually be present in hdfs-site.xml.
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
@@ -5599,7 +5631,7 @@ DriverManager.getConnection(url, props);
The values above need to be reflected in each topology descriptor file deployed to the gateway. The gateway by default includes a sample topology descriptor file {GATEWAY_HOME}/deployments/sandbox.xml. The values in this sample are configured to work with an installed Sandbox VM.
<service>
<role>HDFSUI</role>
- <url>http://sandbox.hortonworks.com:50070/webhdfs</url>
+ <url>http://sandbox.hortonworks.com:50070</url>
</service>
In addition to the service configuration for HDFSUI, the REST service configuration for WEBHDFS is also required.
<service>
Modified: knox/trunk/books/0.14.0/book_service-details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/book_service-details.md?rev=1813985&r1=1813984&r2=1813985&view=diff
==============================================================================
--- knox/trunk/books/0.14.0/book_service-details.md (original)
+++ knox/trunk/books/0.14.0/book_service-details.md Wed Nov 1 19:34:18 2017
@@ -13,7 +13,7 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---->
+-->
## Service Details ##
Modified: knox/trunk/books/0.14.0/book_ui_service_details.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/book_ui_service_details.md?rev=1813985&r1=1813984&r2=1813985&view=diff
==============================================================================
--- knox/trunk/books/0.14.0/book_ui_service_details.md (original)
+++ knox/trunk/books/0.14.0/book_ui_service_details.md Wed Nov 1 19:34:18 2017
@@ -59,7 +59,7 @@ The values in this sample are configured
HDFSUI
- http://sandbox.hortonworks.com:50070/webhdfs
+ http://sandbox.hortonworks.com:50070
In addition to the service configuration for HDFSUI, the REST service configuration for WEBHDFS is also required.
Modified: knox/trunk/books/0.14.0/service_webhdfs.md
URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/service_webhdfs.md?rev=1813985&r1=1813984&r2=1813985&view=diff
==============================================================================
--- knox/trunk/books/0.14.0/service_webhdfs.md (original)
+++ knox/trunk/books/0.14.0/service_webhdfs.md Wed Nov 1 19:34:18 2017
@@ -13,7 +13,7 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---->
+-->
### WebHDFS ###