allura-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kentontay...@apache.org
Subject [1/6] allura git commit: [#7908] docker-compose and .ini files for prod; httpd config proxies back to webapp, sample robots.txt too
Date Mon, 30 Oct 2017 14:42:56 GMT
Repository: allura
Updated Branches:
  refs/heads/master fac64c65a -> 53b18cd86


[#7908] docker-compose and .ini files for prod; httpd config proxies back to webapp, sample
robots.txt too


Project: http://git-wip-us.apache.org/repos/asf/allura/repo
Commit: http://git-wip-us.apache.org/repos/asf/allura/commit/9a35a13e
Tree: http://git-wip-us.apache.org/repos/asf/allura/tree/9a35a13e
Diff: http://git-wip-us.apache.org/repos/asf/allura/diff/9a35a13e

Branch: refs/heads/master
Commit: 9a35a13e322e0e01135a30dc6745678b6a53a2f7
Parents: 0e2b05b
Author: Dave Brondsema <dave@brondsema.net>
Authored: Thu Sep 28 13:25:55 2017 -0400
Committer: Kenton Taylor <ktaylor@slashdotmedia.com>
Committed: Mon Oct 30 10:42:32 2017 -0400

----------------------------------------------------------------------
 Allura/docker-dev.ini                          |   6 +-
 Allura/docs/getting_started/administration.rst |  19 +-
 Allura/docs/getting_started/installation.rst   |  12 +-
 Allura/production-docker-example.ini           | 189 ++++++++++++++++++++
 docker-compose-prod.yml                        | 143 +++++++++++++++
 docker-compose.yml                             |   6 +-
 scm_config/git-http/Dockerfile                 |   4 +-
 scm_config/git-http/git-http.conf              |  22 ++-
 scripts/init-docker-dev.sh                     |   3 +
 9 files changed, 382 insertions(+), 22 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/Allura/docker-dev.ini
----------------------------------------------------------------------
diff --git a/Allura/docker-dev.ini b/Allura/docker-dev.ini
index 709cd79..ee7734f 100644
--- a/Allura/docker-dev.ini
+++ b/Allura/docker-dev.ini
@@ -18,7 +18,7 @@
 [server:main]
 use = egg:Paste#http
 host = 0.0.0.0
-port = 8080
+port = 8088
 
 [app:main]
 use = config:development.ini#main
@@ -29,13 +29,13 @@ ming.main.uri = mongodb://mongo:27017/allura
 ming.project.uri = mongodb://mongo:27017/project-data
 ming.task.uri = mongodb://mongo:27017/task
 
-scm.repos.refresh_base_url = http://web:8080
+scm.repos.refresh_base_url = http://web:8088
 
 scm.repos.root = /allura-data/scm
 
 scm.clonechoices.git = [{"name": "HTTP", "key": "http", "title": "HTTP"}, {"name": "File",
"key": "file", "title": "Filesystem"}]
 ; may need to change "localhost" to your remote host name, or docker-machine IP address
-scm.host.http.git = http://localhost:8081/git$path
+scm.host.http.git = http://localhost:8080/git$path
 scm.host.file.git = /allura-data/scm/git$path
 scm.clonechoices.hg = [{"name": "File", "key": "file", "title": "Filesystem"}]
 scm.host.file.hg = /allura-data/scm/hg$path

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/Allura/docs/getting_started/administration.rst
----------------------------------------------------------------------
diff --git a/Allura/docs/getting_started/administration.rst b/Allura/docs/getting_started/administration.rst
index 9ed2f3d..cb4683c 100644
--- a/Allura/docs/getting_started/administration.rst
+++ b/Allura/docs/getting_started/administration.rst
@@ -84,14 +84,14 @@ Overview
 --------
 
 Allura has many commands and scripts that can be run from the server commandline to
-administrate Allura.  There are also tasks that can be run through the `taskd` system
+administrate Allura.  There are also tasks that can be run through the :code:`taskd` system
 in the background.  These tasks can be submitted via the web at
 http://MYSITE/nf/admin/task_manager  Some paster scripts have been set up
 so that they are runnable as tasks too, giving you the convenience of starting
-them through the web and letting `taskd` execute them, rather than from a server
+them through the web and letting :code:`taskd` execute them, rather than from a server
 shell.
 
-Commands can be discovered and run via the `paster` command when you are in the
+Commands can be discovered and run via the :code:`paster` command when you are in the
 'Allura' directory that has your .ini file.  For example::
 
      paster help
@@ -103,7 +103,7 @@ Commands can be discovered and run via the `paster` command when you are
in the
      paster create-neighborhood development.ini myneighborhood myuser ...
 
 
-Scripts are in the `scripts/` directory and run slightly differently, via `paster script`.
 An extra
+Scripts are in the :file:`scripts/` directory and run slightly differently, via :code:`paster
script`.  An extra
 :kbd:`--` is required to separate script arguments from paster arguments.  Example::
 
      paster script development.ini ../scripts/add_user_to_group.py -- --help
@@ -111,16 +111,21 @@ Scripts are in the `scripts/` directory and run slightly differently,
via `paste
 
      paster script development.ini ../scripts/add_user_to_group.py -- --nbhd /u/ johndoe
Admin
 
-To run these when using docker, prefix with :code:`docker-compose run taskd` and use :code:`docker-dev.ini`
like::
+To run these when using docker, prefix with :code:`docker-compose run taskd` and use :file:`docker-dev.ini`
like::
 
     docker-compose run taskd paster create-neighborhood docker-dev.ini myneighborhood myuser
...
 
+Or with the docker *production* setup::
+
+    docker-compose run --rm oneoff paster create-neighborhood /allura-data/production.ini
myneighborhood myuser ...
+
+
 Tasks can be run via the web interface at http://MYSITE/nf/admin/task_manager  You must know
 the full task name, e.g. :code:`allura.tasks.admin_tasks.install_app`  You can
 optionally provide a username and project and app which will get set on the
-current context (`c`).  You should specify what args and kwargs will be passed
+current context (:kbd:`c`).  You should specify what args and kwargs will be passed
 as parameters to the task.  They are specified in JSON format on the form.  If you are
-running a script via this interface, the `args/kwargs` JSON should be like::
+running a script via this interface, the :kbd:`args/kwargs` JSON should be like::
 
     {
         "args": ["--foo --bar baz"],

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/Allura/docs/getting_started/installation.rst
----------------------------------------------------------------------
diff --git a/Allura/docs/getting_started/installation.rst b/Allura/docs/getting_started/installation.rst
index 1ff1b7d..4b22e1f 100644
--- a/Allura/docs/getting_started/installation.rst
+++ b/Allura/docs/getting_started/installation.rst
@@ -500,13 +500,19 @@ emails sent to that address will be added as comments on the ticket.
 To set up
 
 By default this uses port 8825.  Depending on your mail routing, you may need to change that
port number.
 And if the port is in use, this command will fail.  You can check the log file for any errors.
-To change the port number, edit `development.ini` and change `forgemail.port` to the appropriate
port number for your environment.
+To change the port number, edit :file:`development.ini` and change :samp:`forgemail.port`
to the appropriate port number for your environment.
 
 SMTP in development
 ^^^^^^^^^^^^^^^^^^^
 
-The following command can be used for quick and easy monitoring of smtp during development.
-Just be sure the port matches the `smtp_port` from your `development.ini` (8826 by default).
+The following command can be used for quick and easy monitoring of outgoing email during
development.
+
+.. code-block:: bash
+
+    docker-compose logs -f outmail
+
+If you are running locally without docker, run this command.  Be sure the port matches the
:samp:`smtp_port` from
+your :file:`development.ini` (8826 by default).
 
 .. code-block:: bash
 

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/Allura/production-docker-example.ini
----------------------------------------------------------------------
diff --git a/Allura/production-docker-example.ini b/Allura/production-docker-example.ini
new file mode 100644
index 0000000..f5f918d
--- /dev/null
+++ b/Allura/production-docker-example.ini
@@ -0,0 +1,189 @@
+;       Licensed to the Apache Software Foundation (ASF) under one
+;       or more contributor license agreements.  See the NOTICE file
+;       distributed with this work for additional information
+;       regarding copyright ownership.  The ASF licenses this file
+;       to you under the Apache License, Version 2.0 (the
+;       "License"); you may not use this file except in compliance
+;       with the License.  You may obtain a copy of the License at
+;
+;         http://www.apache.org/licenses/LICENSE-2.0
+;
+;       Unless required by applicable law or agreed to in writing,
+;       software distributed under the License is distributed on an
+;       "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+;       KIND, either express or implied.  See the License for the
+;       specific language governing permissions and limitations
+;       under the License.
+
+
+; Copy this file to /allura-data/production.ini
+;
+; And then you MUST change several settings in here
+;
+; Find every reference of myexamplesite.com here and change it
+; Also change:
+;  site_name
+;  smtp_server
+;  beaker.session.validate_key
+;
+; This file inherits settings from docker-dev.ini and development.ini
+; You are free to make additional changes/additions to this file for other settings
+
+
+[server:main]
+use = egg:Paste#http
+host = 0.0.0.0
+port = 8080
+
+[app:main]
+;  this is a path within docker, thus it's just /allura
+use = config:/allura/Allura/docker-dev.ini#main
+
+; "set" must be used since we're inheriting with the "use" line above
+set debug = false
+
+; Used to uniquify references to static resources, can be a timestamp or any unique value
+; This should be updated each time you deploy (or make significant changes, like new tools,
new css)
+build_key=1276635823
+
+base_url = https://myexamplesite.com
+domain = myexamplesite.com
+site_name = My Example Site
+;force_ssl.pattern = .
+
+; outgoing mail
+; result of `ip addr show docker0`
+; https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach
+smtp_server = 172.17.0.1
+smtp_port = 25
+smtp_tls = true
+
+forgemail.domain = .myexamplesite.com
+forgemail.return_path = noreply@myexamplesite.com
+
+beaker.session.validate_key = 712de83fa0cb0d0f0a383
+
+auth.allow_birth_date = false
+trovecategories.enableediting = admin
+
+scm.repos.refresh_base_url = http://web:8080
+scm.clonechoices.git = [{"name": "HTTPS", "key": "https", "title": "HTTPS"}]
+scm.host.https.git = https://myexamplesite.com/git$path
+;scm.host.rw.git = https://$username@myexamplesite.com/git$path
+scm.repos.tarball.enable = false
+
+jinja_bytecode_cache_type = filesystem
+auto_reload_templates = false
+lcd_timeout = 15
+markdown_cache_threshold = .1
+repo_refs_cache_threshold = .1
+
+stats.sample_rate = .01
+
+
+
+[app:task]
+use = main
+override_root = task ; TurboGears will use controllers/task.py as root controller
+
+
+
+[loggers]
+keys = root, allura, sqlalchemy, paste, pylons, ew, taskdstatus, timermiddleware, tmw_details
+
+[handlers]
+keys = console, stats, taskdstatus, timermiddleware
+
+[formatters]
+keys = generic, stats, timermiddleware
+
+; If you create additional loggers, add them as a key to [loggers]
+[logger_root]
+level = INFO
+handlers = console, stats
+
+[logger_allura]
+level = INFO
+handlers =
+qualname = allura
+
+[logger_sqlalchemy]
+level = INFO
+handlers =
+qualname = sqlalchemy.engine
+; "level = INFO" logs SQL queries.
+; "level = DEBUG" logs SQL queries and results.
+; "level = WARN" logs neither.  (Recommended for production systems.)
+
+[logger_paste]
+level = INFO
+qualname = paste
+handlers =
+
+[logger_pylons]
+level = INFO
+qualname = pylons
+handlers =
+
+[logger_ew]
+; easy widgets
+level = WARN
+qualname = ew
+handlers =
+
+[logger_tmw_details]
+; DEBUG will include every instrumented call in our logging
+level = INFO
+qualname = timermiddleware
+handlers =
+
+[logger_taskdstatus]
+level = INFO
+qualname = taskdstatus
+handlers = taskdstatus
+
+[logger_timermiddleware]
+level = INFO
+handlers = timermiddleware
+qualname = stats
+propagate = 0
+
+; If you create additional handlers, add them as a key to [handlers]
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[handler_stats]
+class = allura.lib.utils.StatsHandler
+args = ('rtstats.log', 'allura', 1)
+level = NOTSET
+formatter = stats
+
+[handler_taskdstatus]
+class = handlers.WatchedFileHandler
+args = ('taskd_status.log', 'a')
+level = NOTSET
+formatter = generic
+
+[handler_timermiddleware]
+class = handlers.WatchedFileHandler
+; if you run 'gunicorn' in allura/Allura/ then that's where this file will be
+; you may want to hard-code a specific directory here.
+args = ('stats.log', 'a')
+level = NOTSET
+formatter = timermiddleware
+
+; If you create additional formatters, add them as a key to [formatters]
+[formatter_generic]
+format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S
+
+[formatter_stats]
+format = %(asctime)s %(created)d %(kwpairs)s
+datefmt = %d/%b/%Y:%H:%M:%S UTC
+
+[formatter_timermiddleware]
+format = {"time": "%(asctime)s,%(msecs)03d", "level": "%(levelname)-5.5s", "name": "%(name)s",
"message": %(message)s}
+datefmt = %Y-%m-%d %H:%M:%S
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/docker-compose-prod.yml
----------------------------------------------------------------------
diff --git a/docker-compose-prod.yml b/docker-compose-prod.yml
new file mode 100644
index 0000000..19bcfaf
--- /dev/null
+++ b/docker-compose-prod.yml
@@ -0,0 +1,143 @@
+#       Licensed to the Apache Software Foundation (ASF) under one
+#       or more contributor license agreements.  See the NOTICE file
+#       distributed with this work for additional information
+#       regarding copyright ownership.  The ASF licenses this file
+#       to you under the Apache License, Version 2.0 (the
+#       "License"); you may not use this file except in compliance
+#       with the License.  You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#       Unless required by applicable law or agreed to in writing,
+#       software distributed under the License is distributed on an
+#       "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+#       KIND, either express or implied.  See the License for the
+#       specific language governing permissions and limitations
+#       under the License.
+
+
+##
+# How to use this file:
+#
+# Change these settings:
+#    VIRTUAL_HOST
+#    LETSENCRYPT_HOST
+#    LETSENCRYPT_EMAIL
+#
+# Copy Allura/production-docker-example.ini to /allura-data/production.ini and review its
contents,
+#  making changes as appropriate
+#
+# In comparision to the development version of docker-compose.yml, this production ready
version:
+#  * only exposes ports that are necessary, limiting them to within docker, or to 127.0.0.1
+#  * sets containers to always restart
+#  * has an nginx proxy to provide HTTPS via letsencrypt.  May take a little time to configure
itself
+#  * has no debugging "outmail" container, emails should go out into the real world
+#  * git-http container serves git and also proxies back to the "web" container
+##
+
+version: "2"
+services:
+  web:
+    build: .
+    environment: &env
+      # PATH=/allura-data/virtualenv/bin:$PATH doesn't work; see https://github.com/docker/compose/issues/650
+      - PATH=/allura-data/virtualenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+    expose:
+      - "8080"
+    volumes: &volumes
+      - .:/allura  # Allura source code from local host
+      - /allura-data:/allura-data  # for virtualenv, scm repos, etc
+    links:
+      - mongo
+      - solr
+    command: gunicorn --paste /allura-data/production.ini
+    restart: always
+
+  taskd:
+    image: allura_web
+    working_dir: /allura/Allura
+    environment: *env
+    command: paster taskd /allura-data/production.ini
+    volumes: *volumes
+    links:
+      - mongo
+      - solr
+    restart: always
+
+  # This is a single-purpose container that does not auto-restart, good for running commands
like:
+  # docker-compose run --rm oneoff paster ensure_index /allura-data/production.ini
+  oneoff:
+    image: allura_web
+    working_dir: /allura/Allura
+    environment: *env
+    volumes: *volumes
+    command: ls /dev/null
+    links:
+      - mongo
+
+  solr:
+    image: solr:6-alpine  # alpine is a very small distro base
+    expose:
+      - "8983"
+    volumes:
+      - ./solr_config/allura:/opt/solr/server/solr/allura
+      - /allura-data/solr:/opt/solr/server/solr/allura/data
+    restart: always
+
+  mongo:
+    image: mongo:3.4
+    ports:
+      - "127.0.0.1:27017:27017"
+    volumes:
+      - /allura-data/mongo:/data/db
+    command: mongod --storageEngine wiredTiger
+    restart: always
+
+  inmail:
+    image: allura_web
+    working_dir: /allura/Allura
+    environment: *env
+    volumes: *volumes
+    command: paster smtp_server /allura-data/production.ini
+    ports:
+      - "127.0.0.1:8825:8825"
+    links:
+      - mongo
+    restart: always
+
+  git-http:
+    build: scm_config/git-http/
+    expose:
+      - "80"
+    volumes: *volumes
+    links:
+      - mongo
+      - web
+    restart: always
+    environment:
+      VIRTUAL_HOST: allura-vm2.apache.org
+      LETSENCRYPT_HOST: allura-vm2.apache.org
+      LETSENCRYPT_EMAIL: dave@brondsema.net
+
+  # References for how we set up the nginx-proxy and letsencrypt-nginx-proxy-companion containers
+  # https://github.com/dataminelab/docker-jenkins-nginx-letsencrypt
+  # https://github.com/dmitrym0/simple-lets-encrypt-docker-compose-sample/blob/master/docker-compose.yml
+  nginx-proxy:
+    image: jwilder/nginx-proxy
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - "./nginx/vhost.d:/etc/nginx/vhost.d"
+      - "./nginx/html:/usr/share/nginx/html"
+      - "./nginx/certs:/etc/nginx/certs"
+      - "/var/run/docker.sock:/tmp/docker.sock:ro"
+    restart: always
+
+  letsencrypt-nginx-proxy-companion:
+    image: jrcs/letsencrypt-nginx-proxy-companion
+    volumes:
+      - "/var/run/docker.sock:/var/run/docker.sock:ro"
+    volumes_from:
+      - "nginx-proxy"
+    restart: always
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker-compose.yml b/docker-compose.yml
index 9374f90..b91a0a0 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -23,7 +23,7 @@ services:
       # PATH=/allura-data/virtualenv/bin:$PATH doesn't work; see https://github.com/docker/compose/issues/650
       - PATH=/allura-data/virtualenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     ports:
-      - "8080:8080"
+      - "8088:8088"
     volumes: &volumes
       - .:/allura  # Allura source code from local host
       - /allura-data:/allura-data  # for virtualenv, scm repos, etc
@@ -78,10 +78,10 @@ services:
     links:
       - mongo
 
-  git-http:
+  http:
     build: scm_config/git-http/
     ports:
-      - "8081:80"
+      - "8080:80"
     volumes: *volumes
     links:
       - mongo

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/scm_config/git-http/Dockerfile
----------------------------------------------------------------------
diff --git a/scm_config/git-http/Dockerfile b/scm_config/git-http/Dockerfile
index b2e1b28..36111b1 100644
--- a/scm_config/git-http/Dockerfile
+++ b/scm_config/git-http/Dockerfile
@@ -40,12 +40,12 @@ ENV APACHE_SERVERNAME localhost
 ENV APACHE_SERVERALIAS docker.localhost
 ENV APACHE_DOCUMENTROOT /var/www
 
+RUN a2enmod cgi proxy proxy_http
+
 ADD ./git-http.conf /etc/apache2/sites-available/
 RUN a2dissite 000-default.conf
 RUN a2ensite git-http.conf
 
-RUN a2enmod cgi
-
 # so that git operations run as root, and can modify the scm repo files
 RUN chmod u+s /usr/lib/git-core/git-http-backend
 

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/scm_config/git-http/git-http.conf
----------------------------------------------------------------------
diff --git a/scm_config/git-http/git-http.conf b/scm_config/git-http/git-http.conf
index f6f059e..6919ed7 100644
--- a/scm_config/git-http/git-http.conf
+++ b/scm_config/git-http/git-http.conf
@@ -21,18 +21,25 @@ ErrorLog /dev/stderr
 
 <VirtualHost *:80>
 
+    # /git/ requests go to the git backend binary
     SetEnv GIT_PROJECT_ROOT /allura-data/scm/git
     SetEnv GIT_HTTP_EXPORT_ALL
     ScriptAlias /git/ /usr/lib/git-core/git-http-backend/
 
     # disable access to default web content
     <Directory /var/www>
-        Order Deny,Allow
-        Deny from all
+        Require all denied
         Options None
         AllowOverride None
     </Directory>
 
+    # any Alias here must be excluded from ProxyPass (at bottom of file)
+    Alias /robots.txt /allura-data/www-misc/robots.txt
+    <Directory /allura-data/www-misc/>
+        Require all granted
+    </Directory>
+
+    # apply security checks to all /git/ requests
     <Location "/git/">
         Require all granted
 
@@ -43,10 +50,17 @@ ErrorLog /dev/stderr
         AuthType Basic
         AuthName "Git Access"
         AuthBasicAuthoritative off
-        PythonOption ALLURA_PERM_URL http://web:8080/auth/repo_permissions
-        PythonOption ALLURA_AUTH_URL http://web:8080/auth/do_login
+        PythonOption ALLURA_PERM_URL http://web:8088/auth/repo_permissions
+        PythonOption ALLURA_AUTH_URL http://web:8088/auth/do_login
         # for 'requests' lib only, doesn't have to be full allura venv
         PythonOption ALLURA_VIRTUALENV /allura-data/virtualenv
     </Location>
 
+    # declare what is being handled here
+    ProxyPass "/robots.txt" !
+    ProxyPass "/git/" !
+    # everything else gets proxied through to the Allura webapp
+    ProxyPass "/" "http://web:8088/"
+    ProxyPassReverse "/" "http://web:8088/"
+
 </VirtualHost>
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/allura/blob/9a35a13e/scripts/init-docker-dev.sh
----------------------------------------------------------------------
diff --git a/scripts/init-docker-dev.sh b/scripts/init-docker-dev.sh
index ddadf08..097c94e 100755
--- a/scripts/init-docker-dev.sh
+++ b/scripts/init-docker-dev.sh
@@ -32,6 +32,9 @@ mkdir -p /allura-data/solr
 echo -e "Changing it's permissions to 777 so that container will have access to it\n"
 chmod 777 /allura-data/solr
 
+mkdir -p /allura-data/www-misc
+echo "# No robots.txt rules here" > /allura-data/www-misc/robots.txt
+
 # share venv to allow update and sharing across containers
 if [ ! -e /allura-data/virtualenv ]; then
     echo -e "Creating virtualenv\n"


Mime
View raw message