whimsical-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Ruby <ru...@apache.org>
Subject [whimsy.git] [1/2] Commit 2118888: LDAP fallover and configuration:
Date Thu, 14 Jan 2016 20:42:02 GMT
Commit 2118888d3b99ed1e23b05ad203e05f2c62096e3b:
    LDAP fallover and configuration:
    - Read multiple LDAP uris from /etc/ldap
    - Select from those hosts randomly
    - Verify that the host is alive
    - Use Puppet configuration as a primary source for configuration information
    - Have configure intelligently update as well as set LDAP configuration values


Branch: refs/heads/master
Author: Sam Ruby <rubys@intertwingly.net>
Committer: Sam Ruby <rubys@intertwingly.net>
Pusher: rubys <rubys@apache.org>

------------------------------------------------------------
README.md                                                    | +++++++ ------
lib/whimsy/asf/ldap.rb                                       | ++++++++ ---
------------------------------------------------------------
204 changes: 141 additions, 63 deletions.
------------------------------------------------------------


diff --git a/README.md b/README.md
index 068e1cf..2b48bb1 100644
--- a/README.md
+++ b/README.md
@@ -24,7 +24,7 @@ Overview
 
 This directory has two main subdirectories...
 
-1. [lib/whimsy/asf](lib/whimsy/asf) contains the "model", i.e., a set of classes 
+1. [lib/whimsy/asf](lib/whimsy/asf) contains the "model", i.e., a set of classes
    which encapsulate access
    to a number of data sources such as LDAP, ICLAs, auth lists, etc.  This
    code originally was developed as a part of separate tools and was later
@@ -34,13 +34,13 @@ This directory has two main subdirectories...
 2. [www](www) contains the "view", largely a set of CGI scripts that produce HTML.
    Generally a CGI script is self contained, including all of the CSS,
    scripts, AJAX logic (client and server), SVG images, etc.  A single script
-   may also produce a set (subtree) of web pages. 
+   may also produce a set (subtree) of web pages.
 
    Some of the directories (like the roster tool) contain [rack](http://rack.github.io/)
    applications.  These can be run independently, or under the Apache web server through
-   the use of [Phusion Passenger](https://www.phusionpassenger.com/). 
-   
-   The board agenda tool is currently hosted [separately](https://github.com/rubys/whimsy-agenda)

+   the use of [Phusion Passenger](https://www.phusionpassenger.com/).
+
+   The board agenda tool is currently hosted [separately](https://github.com/rubys/whimsy-agenda)
    on github, but this will be consolidated into this repository as a part of the effort
    to move whimsy.apache.org to a new VM.
 
@@ -60,11 +60,11 @@ Skip this section if you are running a Docker container or a Vagrant VM.
    'cd' in bash).
 
     For more information:
-    
+
     1. [Understanding Shims](https://github.com/sstephenson/rbenv#understanding-shims)
     2. [Understanding Binstubs](https://github.com/sstephenson/rbenv/wiki/Understanding-binstubs)
     3. [Ruby Version Manager](https://rvm.io/)
-    
+
 
 2. Make sure that the `whimsy-asf` gem installed.  If it is not, run
 
@@ -87,8 +87,8 @@ Skip this section if you are running a Docker container or a Vagrant VM.
 4. Access to LDAP requires configuration, and a cert.
 
  1. The model code determines what host and port to connect to by parsing
-      either `/etc/ldap/ldap.conf` or `/etc/ldap/ldap.conf` for a line that looks
-      like the following:
+      either `/etc/ldap/ldap.conf` or `/etc/ldap/ldap.conf` for a line that
+      looks like the following:
         `uri     ldaps://ldap1-us-east.apache.org:636`
 
  2. A `TLS_CACERT` can be obtained via either of the following commands:
@@ -96,22 +96,26 @@ Skip this section if you are running a Docker container or a Vagrant VM.
         `ruby -r whimsy/asf -e "puts ASF::LDAP.cert"`<br/>
         `openssl s_client -connect ldap1-us-east.apache.org:636 </dev/null`
 
-      Copy from `BEGIN` to `END` inclusive into the file 
+      Copy from `BEGIN` to `END` inclusive into the file
       `/etc/ldap/asf-ldap-client.pem`.  Point to the file in
       `/etc/ldap/ldap.conf` with a line like the following:
 
      ```   TLS_CACERT      /etc/ldap/asf-ldap-client.pem```
 
-      N.B. OpenLDAP uses `/etc/openldap/` instead of `/etc/ldap/` 
-      Adjust the paths above as necessary
+      N.B. OpenLDAP on Mac OS/X uses `/etc/openldap/` instead of `/etc/ldap/`
+      Adjust the paths above as necessary.  Additionally ensure that
+      that `TLS_REQCERT` is set to `allow`.
 
-      Note: the certificate is needed because the LDAP hosts use a self-signed
-      certificate
-      
-   These above updates can be done for you with the following command:
+      Note: the certificate is needed because the ASF LDAP hosts use a
+      self-signed certificate.
+
+   All these updates can be done for you with the following command:
 
         sudo ruby -r whimsy/asf -e "ASF::LDAP.configure"
 
+   These above command can also be used to update your configuration as
+   the ASF changes LDAP servers.
+
 5. Verify that the configuration is correct by running:
 
    `ruby examples/board.rb`
@@ -121,12 +125,12 @@ Skip this section if you are running a Docker container or a Vagrant
VM.
 Further Reading
 ===============
 
-An [example](https://github.com/rubys/whimsy-agenda#readme) of a complete tool 
-that makes full use of the library factoring, has a suite of test cases, and 
-client componentization (using ReactJS), and provides instructions for setting 
+An [example](https://github.com/rubys/whimsy-agenda#readme) of a complete tool
+that makes full use of the library factoring, has a suite of test cases, and
+client componentization (using ReactJS), and provides instructions for setting
 up both a Docker component and a Vagrant VM:
 
 If you would like to understand how the view code works, you can get started
-by looking at a 
-few of the [Wunderbar demos](https://github.com/rubys/wunderbar/tree/master/demo) 
+by looking at a few of the
+[Wunderbar demos](https://github.com/rubys/wunderbar/tree/master/demo)
 and [README](https://github.com/rubys/wunderbar/blob/master/README.md).
diff --git a/lib/whimsy/asf/ldap.rb b/lib/whimsy/asf/ldap.rb
index 6b6bae8..33948c7 100644
--- a/lib/whimsy/asf/ldap.rb
+++ b/lib/whimsy/asf/ldap.rb
@@ -1,6 +1,7 @@
 require 'wunderbar'
 require 'ldap'
 require 'weakref'
+require 'net/http'
 
 module ASF
   module LDAP
@@ -14,6 +15,68 @@ module LDAP
       ldaps://snappy5.apache.org:636
       ldaps://ldap2-lw-us.apache.org:636
     )
+
+    # fetch configuration from apache/infrastructure-puppet
+    def self.puppet_config
+      return @puppet if @puppet
+      file = '/apache/infrastructure-puppet/deployment/data/common.yaml'
+      http = Net::HTTP.new('raw.githubusercontent.com', 443)
+      http.use_ssl = true
+      @puppet = YAML.load(http.request(Net::HTTP::Get.new(file)).body)
+    end
+
+    # extract the ldapcert from the puppet configuration
+    def self.puppet_cert
+      puppet_config['ldapclient::ldapcert']
+    end
+
+    # extract the ldap servers from the puppet configuration
+    def self.puppet_ldapservers
+      puppet_config['ldapserver::slapd_peers'].values.
+        map {|host| "ldaps://#{host}:636"}
+    rescue
+      nil
+    end
+
+    # connect to LDAP
+    def self.connect
+      hosts.shuffle.each do |host|
+        Wunderbar.info "Connecting to LDAP server: #{host}"
+
+        begin
+          # request connection
+          uri = URI.parse(host)
+          if uri.scheme == 'ldaps'
+            ldap = ::LDAP::SSLConn.new(uri.host, uri.port)
+          else
+            ldap = ::LDAP::Conn.new(uri.host, uri.port)
+          end
+
+          # test the connection
+          ldap.bind
+          ldap.unbind rescue nil
+
+          # save the host
+          @host = host
+
+          return ldap
+        rescue ::LDAP::ResultError => re
+          Wunderbar.error "Error connecting to LDAP server #{host}: " +
+            re.message
+        end
+
+        return nil
+      end
+    end
+  end
+
+  # backwards compatibility for tools that called this interface, and
+  # a part of the refresh strategy (something that should be revisited
+  # with WeakReferences instead).
+  def self.init_ldap
+    return @ldap if @ldap
+    @mtime = Time.now
+    @ldap = ASF::LDAP.init
   end
 
   # determine where ldap.conf resides
@@ -23,27 +86,6 @@ module LDAP
     ETCLDAP = '/etc/ldap'
   end
 
-  # determine whether or not the LDAP API can be used
-  def self.init_ldap
-    @ldap = nil
-    @mtime = Time.now
-
-    host = ASF::LDAP.host
-
-    Wunderbar.info "Connecting to LDAP server: #{host}"
-
-    begin
-      uri = URI.parse(host)
-      if uri.scheme == 'ldaps'
-        @ldap = ::LDAP::SSLConn.new(uri.host, uri.port)
-      else
-        @ldap = ::LDAP::Conn.new(uri.host, uri.port)
-      end
-    rescue ::LDAP::ResultError=>re
-      Wunderbar.error "Error binding to LDAP server: message: ["+ re.message + "]"
-    end
-  end
-
   def self.ldap
     @ldap || self.init_ldap
   end
@@ -355,59 +397,91 @@ def add(people)
   module LDAP
     def self.bind(user, password, &block)
       dn = ASF::Person.new(user).dn
+      ASF.ldap.unbind rescue nil
       if block
         ASF.ldap.bind(dn, password, &block)
+        ASF.init_ldap
       else
         ASF.ldap.bind(dn, password)
       end
-      ASF.init_ldap
     end
 
-    # select LDAP host
-    def self.host
+    # determine what LDAP hosts are available
+    def self.hosts
       # try whimsy config
-      host = ASF::Config.get(:ldap)
+      hosts = Array(ASF::Config.get(:ldap))
 
       # check system configuration
-      unless host
+      if hosts.empty?
         conf = "#{ETCLDAP}/ldap.conf"
         if File.exist? conf
-          host = File.read(conf)[/^uri\s+(ldaps?:\/\/\S+?:\d+)/i, 1]
+          uris = File.read(conf)[/^uri\s+(.*)/i, 1].to_s
+          hosts = uris.scan(/ldaps?:\/\/\S+?:\d+/)
         end
       end
 
-      # if all else fails, pick one at random
-      host = ASF::LDAP::HOSTS.sample unless host
+      # if all else fails, use default list
+      hosts = ASF::LDAP::HOSTS if hosts.empty?
+
+      hosts
+    end
 
-      host
+    # select LDAP host
+    def self.host
+      @host ||= hosts.sample
     end
 
     # query and extract cert from openssl output
-    def self.cert
+    def self.extract_cert
       host = LDAP.host[%r{//(.*?)(/|$)}, 1]
-      query = "openssl s_client -connect #{host} -showcerts"
-      output = `#{query} < /dev/null 2> /dev/null`
-      output[/^-+BEGIN.*?\n-+END[^\n]+\n/m]
+      puts ['openssl', 's_client', '-connect', host, '-showcerts'].join(' ')
+      out, err, rc = Open3.capture3 'openssl', 's_client',
+        '-connect', host, '-showcerts'
+      out[/^-+BEGIN.*?\n-+END[^\n]+\n/m]
     end
 
     # update /etc/ldap.conf. Usage:
+    #
     #   sudo ruby -r whimsy/asf -e "ASF::LDAP.configure"
+    #
     def self.configure
-      if not File.exist? "#{ETCLDAP}/asf-ldap-client.pem"
-        File.write "#{ETCLDAP}/asf-ldap-client.pem", self.cert
+      cert = Dir["#{ETCLDAP}/asf*-ldap-client.pem"].first
+
+      # verify/obtain/write the cert
+      if not cert
+        cert = "#{ETCLDAP}/asf-ldap-client.pem"
+        File.write cert, ASF::LDAP.puppet_cert || self.extract_cert
       end
 
+      # read the current configuration file
       ldap_conf = "#{ETCLDAP}/ldap.conf"
       content = File.read(ldap_conf)
-      unless content.include? 'asf-ldap-client.pem'
-        content.gsub!(/^TLS_CACERT/, '# TLS_CACERT')
-        content.gsub!(/^TLS_REQCERT/, '# TLS_REQCERT')
+
+      # ensure that the right cert is used
+      unless content =~ /asf.*-ldap-client\.pem/
+        content.gsub!(/^TLS_CACERT/i, '# TLS_CACERT')
         content += "TLS_CACERT #{ETCLDAP}/asf-ldap-client.pem\n"
-        content += "uri #{LDAP.host}\n"
+      end
+
+      # provide the URIs of the ldap hosts
+      content.gsub!(/^URI/, '# URI')
+      content += "uri \n" unless content =~ /^uri /
+      content[/uri (.*)\n/, 1] = hosts.join(' ')
+
+      # verify/set the base
+      unless content.include? 'base dc=apache'
+        content.gsub!(/^BASE/i, '# BASE')
         content += "base dc=apache,dc=org\n"
-        content += "TLS_REQCERT allow\n" if ETCLDAP.include? 'openldap'
-        File.write(ldap_conf, content)
       end
+
+      # ensure TLS_REQCERT is allow (Mac OS/X only)
+      if ETCLDAP.include? 'openldap' and not content.include? 'REQCERT allow'
+        content.gsub!(/^TLS_REQCERT/i, '# TLS_REQCERT')
+        content += "TLS_REQCERT allow\n"
+      end
+
+      # write the configuration if there were any changes
+      File.write(ldap_conf, content) unless content == File.read(ldap_conf)
     end
   end
 end

Mime
View raw message