storm-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo...@apache.org
Subject [11/50] storm git commit: STORM-349. (Security) ui actions should have nimbus like authroization. Upgraded ring to 1.3.0. Added IAuthorizer instance to UI daemon. Added hadoop-auth as a dependency which allow users to configure AuthenticationFilter.
Date Thu, 13 Nov 2014 19:37:06 GMT
STORM-349. (Security) ui actions should have nimbus like authroization.
Upgraded ring to 1.3.0.  Added IAuthorizer instance to UI daemon. Added
hadoop-auth as a dependency which allow users to configure
AuthenticationFilter.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/3208a8e3
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/3208a8e3
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/3208a8e3

Branch: refs/heads/master
Commit: 3208a8e3f39140227fe6be9c686b795a19777e64
Parents: fa40fbc
Author: Sriharsha Chintalapani <mail@harsha.io>
Authored: Mon Aug 4 19:18:26 2014 -0700
Committer: Sriharsha Chintalapani <mail@harsha.io>
Committed: Mon Aug 4 19:18:26 2014 -0700

----------------------------------------------------------------------
 SECURITY.md                                     |  25 ++++-
 pom.xml                                         |   8 +-
 storm-core/pom.xml                              |  21 +++-
 storm-core/src/clj/backtype/storm/ui/core.clj   | 105 +++++++++----------
 .../src/clj/backtype/storm/ui/helpers.clj       |  15 +--
 storm-core/test/clj/backtype/storm/ui_test.clj  |  49 ---------
 6 files changed, 103 insertions(+), 120 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/SECURITY.md
----------------------------------------------------------------------
diff --git a/SECURITY.md b/SECURITY.md
index 882954a..f919de7 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -54,7 +54,26 @@ must have it set to the actual port that they are going to bind to.
 
 The servlet filters are prefered because it allows indavidual topologies to
 specificy who is and who is not allowed to access the pages associated with
-them.
+them.  
+
+Storm UI can be configured to use AuthenticationFilter from hadoop-auth.
+```yaml
+ui.filter: "org.apache.hadoop.security.authentication.server.AuthenticationFilter"
+ui.filter.params:
+   "type": "kerberos"
+   "kerberos.principal": "HTTP/nimbus.witzend.com"
+   "kerberos.keytab": "/vagrant/keytabs/http.keytab"
+   "kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
+```
+make sure to create a prinicpal 'HTTP/{hostname}' (here hostname should be the one where
UI daemon runs
+
+Once configured users needs to do kinit before accessing UI.
+Ex:
+curl  -i --negotiate -u:anyUser  -b ~/cookiejar.txt -c ~/cookiejar.txt  http://storm-ui-hostname:8080/api/v1/cluster/summary
+
+1) Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click
to  add value "http://storm-ui-hostname:8080"
+2) Google-chrome:  start from command line with: google-chrome --auth-server-whitelist="*storm-ui-hostname"
--auth-negotiate-delegate-whitelist="*storm-ui-hostname"   
+3) IE:  Configure trusted websites to include "storm-ui-hostname" and allow negotiation for
that website 
 
 ## Authentication (Kerberos)
 
@@ -335,7 +354,3 @@ The Logviewer deamon now is also responsible for cleaning up old log files
for d
 
 ### DRPC
 Hopefully more on this soon
-
-
-
-  

http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 87c52ea..04b6ef1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -187,7 +187,8 @@
         <clj-time.version>0.4.1</clj-time.version>
         <curator.version>2.4.0</curator.version>
         <json-simple.version>1.1</json-simple.version>
-        <ring.version>0.3.11</ring.version>
+        <ring.version>1.3.0</ring.version>
+        <jetty.version>7.6.13.v20130916</jetty.version>
         <clojure.tools.logging.version>0.2.3</clojure.tools.logging.version>
         <clojure.math.numeric-tower.version>0.0.1</clojure.math.numeric-tower.version>
         <carbonite.version>1.4.0</carbonite.version>
@@ -390,6 +391,11 @@
                 <version>${ring.version}</version>
             </dependency>
             <dependency>
+	      <groupId>org.eclipse.jetty</groupId>
+	      <artifactId>jetty-servlet</artifactId>
+	      <version>${jetty.version}</version>
+            </dependency>
+            <dependency>
                 <groupId>org.clojure</groupId>
                 <artifactId>tools.logging</artifactId>
                 <version>${clojure.tools.logging.version}</version>

http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/storm-core/pom.xml
----------------------------------------------------------------------
diff --git a/storm-core/pom.xml b/storm-core/pom.xml
index c677b7c..3ac8f23 100644
--- a/storm-core/pom.xml
+++ b/storm-core/pom.xml
@@ -60,6 +60,10 @@
             <artifactId>ring-jetty-adapter</artifactId>
         </dependency>
         <dependency>
+            <groupId>org.eclipse.jetty</groupId>
+            <artifactId>jetty-servlet</artifactId>
+        </dependency>
+        <dependency>
             <groupId>org.clojure</groupId>
             <artifactId>tools.logging</artifactId>
         </dependency>
@@ -96,7 +100,7 @@
             <artifactId>data.codec</artifactId>
             <scope>test</scope>
         </dependency>
- 
+
         <!--java-->
         <dependency>
             <groupId>commons-io</groupId>
@@ -197,6 +201,21 @@
             <scope>test</scope>
         </dependency>
         <dependency>
+	  <groupId>org.apache.hadoop</groupId>
+	  <artifactId>hadoop-auth</artifactId>
+	  <version>2.4.0</version>
+          <exclusions>
+            <exclusion>
+              <groupId>log4j</groupId>
+              <artifactId>log4j</artifactId>
+            </exclusion>
+            <exclusion>
+              <groupId>org.slf4j</groupId>
+              <artifactId>slf4j-log4j12</artifactId>
+            </exclusion>
+          </exclusions>
+        </dependency>
+        <dependency>
         	<groupId>junit</groupId>
         	<artifactId>junit</artifactId>
         	<version>4.1</version>

http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/storm-core/src/clj/backtype/storm/ui/core.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/ui/core.clj b/storm-core/src/clj/backtype/storm/ui/core.clj
index 8dd301e..0aea7ec 100644
--- a/storm-core/src/clj/backtype/storm/ui/core.clj
+++ b/storm-core/src/clj/backtype/storm/ui/core.clj
@@ -20,8 +20,8 @@
   (:use [hiccup core page-helpers])
   (:use [backtype.storm config util log])
   (:use [backtype.storm.ui helpers])
-  (:use [backtype.storm.daemon [common :only [ACKER-COMPONENT-ID ACKER-INIT-STREAM-ID
-                                              ACKER-ACK-STREAM-ID ACKER-FAIL-STREAM-ID system-id?]]])
+  (:use [backtype.storm.daemon [common :only [ACKER-COMPONENT-ID ACKER-INIT-STREAM-ID ACKER-ACK-STREAM-ID
+                                              ACKER-FAIL-STREAM-ID system-id? mk-authorization-handler]]])
   (:use [ring.adapter.jetty :only [run-jetty]])
   (:use [clojure.string :only [blank? lower-case trim]])
   (:import [backtype.storm.utils Utils])
@@ -30,6 +30,8 @@
             ErrorInfo ClusterSummary SupervisorSummary TopologySummary
             Nimbus$Client StormTopology GlobalStreamId RebalanceOptions
             KillOptions])
+  (:import [backtype.storm.security.auth AuthUtils ReqContext])
+  (:import [backtype.storm.generated AuthorizationException])
   (:import [backtype.storm.security.auth AuthUtils])
   (:import [java.io File])
   (:require [compojure.route :as route]
@@ -40,6 +42,9 @@
   (:gen-class))
 
 (def ^:dynamic *STORM-CONF* (read-storm-config))
+(def ^:dynamic *UI-ACL-HANDLER* (mk-authorization-handler (*STORM-CONF* NIMBUS-AUTHORIZER)
*STORM-CONF*))
+
+(def http-creds-handler (AuthUtils/GetUiHttpCredentialsPlugin *STORM-CONF*))
 
 (defmacro with-nimbus
   [nimbus-sym & body]
@@ -47,33 +52,19 @@
      [~nimbus-sym (*STORM-CONF* NIMBUS-HOST) (*STORM-CONF* NIMBUS-THRIFT-PORT)]
      ~@body))
 
-(defn authorized-ui-user?
-  [user conf topology-conf]
-  (let [ui-users (concat (conf UI-USERS)
-                         (conf NIMBUS-ADMINS)
-                         (topology-conf UI-USERS)
-                         (topology-conf TOPOLOGY-USERS))]
-    (or (blank? (conf UI-FILTER))
-        (and (not (blank? user))
-          (some #(= % user) ui-users)))))
-
-(defn assert-authorized-ui-user
-  [user conf topology-conf]
-  (if (not (authorized-ui-user? user conf topology-conf))
-    ;;TODO need a better exception here so the UI can appear better
-    (throw (RuntimeException. (str "User " user " is not authorized.")))))
-
-(defn- ui-actions-enabled?
-  []
-  (= "true" (lower-case (*STORM-CONF* UI-ACTIONS-ENABLED))))
-
-(defn assert-authorized-topology-user
-  [user]
-  ;;TODO eventually we will want to use the Authorizatin handler from nimbus, but for now
-  ;; Disable the calls conditionally
-  (if (not (ui-actions-enabled?))
-    ;;TODO need a better exception here so the UI can appear better
-    (throw (RuntimeException. (str "Topology actions for the UI have been disabled")))))
+(defn assert-authorized-user
+  ([servlet-request op]
+    (assert-authorized-user servlet-request op nil))
+  ([servlet-request op topology-conf]
+     (if http-creds-handler (.populateContext http-creds-handler (ReqContext/context) servlet-request))
+     (if *UI-ACL-HANDLER*
+       (let [context (ReqContext/context)]
+         (if-not (.permit *UI-ACL-HANDLER* context op topology-conf)
+           (let [principal (.principal context)
+                 user (if principal (.getName principal) "unknown")]
+             (throw (AuthorizationException.
+                     (str "UI request '" op "' for '"
+                          user "' user is not authorized")))))))))
 
 (defn get-filled-stats
   [summs]
@@ -474,9 +465,9 @@
                                  :checked (is-ack-stream (get m :stream))}))))))]
     (map (fn [row]
            {:row row}) (partition 4 4 nil streams))))
-  
+
 (defn mk-visualization-data
-  [id window include-sys? user]
+  [id window include-sys?]
   (with-nimbus
     nimbus
     (let [window (if window window ":all-time")
@@ -491,10 +482,9 @@
           bolt-comp-summs (group-by-comp bolt-summs)
           bolt-comp-summs (filter-key (mk-include-sys-fn include-sys?)
                                       bolt-comp-summs)
-          topology-conf (from-json 
+          topology-conf (from-json
                           (.getTopologyConf ^Nimbus$Client nimbus id))]
-      (assert-authorized-ui-user user *STORM-CONF* topology-conf)
-      (visualization-data 
+      (visualization-data
        (merge (hashmap-to-persistent spouts)
               (hashmap-to-persistent bolts))
        spout-comp-summs bolt-comp-summs window id))))
@@ -675,7 +665,6 @@
                                               bolt-comp-summs
                                               window
                                               id)]
-      (assert-authorized-ui-user user *STORM-CONF* topology-conf)
       (merge
        (topology-summary summ)
        {"user" user
@@ -686,8 +675,7 @@
         "spouts" (spout-comp id spout-comp-summs (.get_errors summ) window include-sys?)
         "bolts" (bolt-comp id bolt-comp-summs (.get_errors summ) window include-sys?)
         "configuration" topology-conf
-        "visualizationTable" (stream-boxes visualizer-data)
-        "uiActionsEnabled" (ui-actions-enabled?)}))))
+        "visualizationTable" (stream-boxes visualizer-data)}))))
 
 (defn spout-output-stats
   [stream-summary window]
@@ -842,7 +830,6 @@
           spec (cond (= type :spout) (spout-stats window summ component summs include-sys?)
                      (= type :bolt) (bolt-stats window summ component summs include-sys?))
           errors (component-errors (get (.get_errors summ) component) topology-id)]
-      (assert-authorized-ui-user user *STORM-CONF* topology-conf)
       (merge
         {"user" user
          "id" component
@@ -855,6 +842,10 @@
          "windowHint" (window-hint window)}
        spec errors))))
 
+(defn topology-config [topology-id]
+  (with-nimbus nimbus
+     (from-json (.getTopologyConf ^Nimbus$Client nimbus topology-id))))
+
 (defn check-include-sys?
   [sys?]
   (if (or (nil? sys?) (= "false" sys?)) false true))
@@ -871,42 +862,44 @@
        (cluster-configuration))
   (GET "/api/v1/cluster/summary" [:as {:keys [cookies servlet-request]}]
        (let [user (.getUserName http-creds-handler servlet-request)]
+         (assert-authorized-user servlet-request "getClusterInfo")
          (json-response (cluster-summary user))))
-  (GET "/api/v1/supervisor/summary" []
+  (GET "/api/v1/supervisor/summary" [:as {:keys [cookies servlet-request]}]
+       (assert-authorized-user servlet-request "getClusterInfo")
        (json-response (supervisor-summary)))
-  (GET "/api/v1/topology/summary" []
+  (GET "/api/v1/topology/summary" [:as {:keys [cookies servlet-request]}]
+       (assert-authorized-user servlet-request "getClusterInfo")
        (json-response (all-topologies-summary)))
   (GET  "/api/v1/topology/:id" [:as {:keys [cookies servlet-request]} id & m]
         (let [id (url-decode id)
               user (.getUserName http-creds-handler servlet-request)]
+          (assert-authorized-user servlet-request "getTopology" (topology-config id))
           (json-response (topology-page id (:window m) (check-include-sys? (:sys m)) user))))
   (GET "/api/v1/topology/:id/visualization" [:as {:keys [cookies servlet-request]} id &
m]
-        (let [id (url-decode id)
-              user (.getUserName http-creds-handler servlet-request)]
-          (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys
m)) user))))
+        (let [id (url-decode id)]
+          (assert-authorized-user servlet-request "getTopology" (topology-config id))
+          (json-response (mk-visualization-data id (:window m) (check-include-sys? (:sys
m))))))
   (GET "/api/v1/topology/:id/component/:component" [:as {:keys [cookies servlet-request]}
id component & m]
        (let [id (url-decode id)
              component (url-decode component)
              user (.getUserName http-creds-handler servlet-request)]
+         (assert-authorized-user servlet-request "getTopology" (topology-config id))
          (json-response (component-page id component (:window m) (check-include-sys? (:sys
m)) user))))
   (POST "/api/v1/topology/:id/activate" [:as {:keys [cookies servlet-request]} id]
     (with-nimbus nimbus
       (let [id (url-decode id)
             tplg (.getTopologyInfo ^Nimbus$Client nimbus id)
-            name (.get_name tplg)
-            user (.getUserName http-creds-handler servlet-request)]
-        (assert-authorized-topology-user user)
+            name (.get_name tplg)]
+        (assert-authorized-user servlet-request "activate" (topology-config id))
         (.activate nimbus name)
         (log-message "Activating topology '" name "'")))
     (resp/redirect (str "/api/v1/topology/" id)))
-
   (POST "/api/v1/topology/:id/deactivate" [:as {:keys [cookies servlet-request]} id]
     (with-nimbus nimbus
       (let [id (url-decode id)
             tplg (.getTopologyInfo ^Nimbus$Client nimbus id)
-            name (.get_name tplg)
-            user (.getUserName http-creds-handler servlet-request)]
-        (assert-authorized-topology-user user)
+            name (.get_name tplg)]
+        (assert-authorized-user servlet-request "deactivate" (topology-config id))
         (.deactivate nimbus name)
         (log-message "Deactivating topology '" name "'")))
     (resp/redirect (str "/api/v1/topology/" id)))
@@ -915,9 +908,8 @@
       (let [id (url-decode id)
             tplg (.getTopologyInfo ^Nimbus$Client nimbus id)
             name (.get_name tplg)
-            options (RebalanceOptions.)
-            user (.getUserName http-creds-handler servlet-request)]
-        (assert-authorized-topology-user user)
+            options (RebalanceOptions.)]
+        (assert-authorized-user servlet-request "rebalance" (topology-config id))
         (.set_wait_secs options (Integer/parseInt wait-time))
         (.rebalance nimbus name options)
         (log-message "Rebalancing topology '" name "' with wait time: " wait-time " secs")))
@@ -927,9 +919,8 @@
       (let [id (url-decode id)
             tplg (.getTopologyInfo ^Nimbus$Client nimbus id)
             name (.get_name tplg)
-            options (KillOptions.)
-            user (.getUserName http-creds-handler servlet-request)]
-        (assert-authorized-topology-user user)
+            options (KillOptions.)]
+        (assert-authorized-user servlet-request "killTopology" (topology-config id))
         (.set_wait_secs options (Integer/parseInt wait-time))
         (.killTopologyWithOpts nimbus name options)
         (log-message "Killing topology '" name "' with wait time: " wait-time " secs")))
@@ -972,7 +963,7 @@
                           :join? false
                           :configurator (fn [server]
                                           (doseq [connector (.getConnectors server)]
-                                            (.setHeaderBufferSize connector header-buffer-size))
+                                            (.setRequestHeaderSize connector header-buffer-size))
                                           (config-filter server app filters-confs))}))
    (catch Exception ex
      (log-error ex))))

http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/storm-core/src/clj/backtype/storm/ui/helpers.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/backtype/storm/ui/helpers.clj b/storm-core/src/clj/backtype/storm/ui/helpers.clj
index 60e7bd2..8033c50 100644
--- a/storm-core/src/clj/backtype/storm/ui/helpers.clj
+++ b/storm-core/src/clj/backtype/storm/ui/helpers.clj
@@ -23,7 +23,8 @@
   (:use [backtype.storm.util :only [clojurify-structure uuid defnk url-encode]])
   (:use [clj-time coerce format])
   (:import [backtype.storm.generated ExecutorInfo ExecutorSummary])
-  (:import [org.mortbay.jetty.security SslSocketConnector])
+  (:import [org.eclipse.jetty.server.ssl SslSocketConnector]
+           [org.eclipse.jetty.servlet ServletHolder FilterMapping])
   (:require [ring.util servlet])
   (:require [compojure.route :as route]
             [compojure.handler :as handler]))
@@ -129,7 +130,7 @@ $(\"table#%s\").each(function(i) { $(this).tablesorter({ sortList: %s,
headers:
      )))
 
 (defn url-format [fmt & args]
-  (String/format fmt 
+  (String/format fmt
     (to-array (map #(url-encode (str %)) args))))
 
 (defn to-tasks [^ExecutorInfo e]
@@ -166,18 +167,18 @@ $(\"table#%s\").each(function(i) { $(this).tablesorter({ sortList: %s,
headers:
 
 (defn config-filter [server handler filters-confs]
   (if filters-confs
-    (let [servlet-holder (org.mortbay.jetty.servlet.ServletHolder.
+    (let [servlet-holder (ServletHolder.
                            (ring.util.servlet/servlet handler))
-          context (doto (org.mortbay.jetty.servlet.Context. server "/")
+          context (doto (org.eclipse.jetty.servlet.ServletContextHandler. server "/")
                     (.addServlet servlet-holder "/"))]
       (doseq [{:keys [filter-name filter-class filter-params]} filters-confs]
         (if filter-class
-          (let [filter-holder (doto (org.mortbay.jetty.servlet.FilterHolder.)
+          (let [filter-holder (doto (org.eclipse.jetty.servlet.FilterHolder.)
                                 (.setClassName filter-class)
                                 (.setName (or filter-name filter-class))
                                 (.setInitParameters (or filter-params {})))]
-            (.addFilter context filter-holder "/*" org.mortbay.jetty.Handler/ALL))))
-      (.addHandler server context))))
+            (.addFilter context filter-holder "/*" FilterMapping/ALL))))
+      (.setHandler server context))))
 
 (defn ring-response-from-exception [ex]
   {:headers {}

http://git-wip-us.apache.org/repos/asf/storm/blob/3208a8e3/storm-core/test/clj/backtype/storm/ui_test.clj
----------------------------------------------------------------------
diff --git a/storm-core/test/clj/backtype/storm/ui_test.clj b/storm-core/test/clj/backtype/storm/ui_test.clj
deleted file mode 100644
index 21ddb33..0000000
--- a/storm-core/test/clj/backtype/storm/ui_test.clj
+++ /dev/null
@@ -1,49 +0,0 @@
-;; Licensed to the Apache Software Foundation (ASF) under one
-;; or more contributor license agreements.  See the NOTICE file
-;; distributed with this work for additional information
-;; regarding copyright ownership.  The ASF licenses this file
-;; to you under the Apache License, Version 2.0 (the
-;; "License"); you may not use this file except in compliance
-;; with the License.  You may obtain a copy of the License at
-;;
-;; http://www.apache.org/licenses/LICENSE-2.0
-;;
-;; Unless required by applicable law or agreed to in writing, software
-;; distributed under the License is distributed on an "AS IS" BASIS,
-;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-;; See the License for the specific language governing permissions and
-;; limitations under the License.
-(ns backtype.storm.ui-test
-  (:use [clojure test])
-  (:use [backtype.storm config])
-  (:use [backtype.storm testing])
-  (:require [backtype.storm.ui [core :as core]])
-  )
-
-(deftest test-authorized-ui-user
-  (testing "allow cluster admin"
-    (let [conf {UI-FILTER "something" NIMBUS-ADMINS ["alice"]}]
-      (is (core/authorized-ui-user? "alice" conf {}))))
-
-  (testing "ignore any cluster-set topology.users"
-    (let [conf {UI-FILTER "something" TOPOLOGY-USERS ["alice"]}]
-      (is (not (core/authorized-ui-user? "alice" conf {})))))
-
-  (testing "allow cluster ui user"
-    (let [conf {UI-FILTER "something" UI-USERS ["alice"]}]
-      (is (core/authorized-ui-user? "alice" conf {}))))
-
-  (testing "allow submitted topology user"
-    (let [topo-conf {TOPOLOGY-USERS ["alice"]}]
-      (is (core/authorized-ui-user? "alice" {UI-FILTER "something"} topo-conf))))
-
-  (testing "allow submitted ui user"
-    (let [topo-conf {UI-USERS ["alice"]}]
-      (is (core/authorized-ui-user? "alice" {UI-FILTER "something"} topo-conf))))
-
-  (testing "disallow user not in nimbus admin, topo user, or ui user"
-    (is (not (core/authorized-ui-user? "alice" {UI-FILTER "something"} {}))))
-
-  (testing "user cannot override nimbus admin"
-    (let [topo-conf {NIMBUS-ADMINS ["alice"]}]
-      (is (not (core/authorized-ui-user? "alice" {UI-FILTER "something"} topo-conf))))))


Mime
View raw message