yetus-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject yetus git commit: YETUS-570. Report and optionally kill stale JVMs between unit test modules
Date Fri, 03 Nov 2017 03:14:10 GMT
Repository: yetus
Updated Branches:
  refs/heads/master 834d7a470 -> 1dcbbeb3f


YETUS-570. Report and optionally kill stale JVMs between unit test modules

Signed-off-by: Sean Busbey <busbey@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/yetus/repo
Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/1dcbbeb3
Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/1dcbbeb3
Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/1dcbbeb3

Branch: refs/heads/master
Commit: 1dcbbeb3f3ecac50f08272daba8aa196dd023227
Parents: 834d7a4
Author: Allen Wittenauer <aw@apache.org>
Authored: Tue Oct 24 09:46:01 2017 -0700
Committer: Allen Wittenauer <aw@apache.org>
Committed: Thu Nov 2 20:13:40 2017 -0700

----------------------------------------------------------------------
 .../in-progress/precommit-advanced.md           |  17 +-
 .../in-progress/precommit-basic.md              |  10 +-
 precommit/core.d/01-common.sh                   |  44 ++++
 precommit/core.d/docker.sh                      |  29 +--
 precommit/core.d/reaper.sh                      | 252 +++++++++++++++++++
 precommit/test-patch.d/maven.sh                 |   3 +
 precommit/test-patch.sh                         | 160 +++++++++++-
 7 files changed, 473 insertions(+), 42 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/asf-site-src/source/documentation/in-progress/precommit-advanced.md
----------------------------------------------------------------------
diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
index 159e154..bdfe5fe 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
@@ -21,6 +21,7 @@ test-patch
 ==========
 
 * [Docker Support](#docker-support)
+* [Process Reaper](#test-reaper)
 * [Plug-ins](#plug-ins)
 * [Personalities](#personalities)
 * [Important Variables](#important-variables)
@@ -43,13 +44,21 @@ For example, `--dockeronfail=continue` means if the Dockerfile can't be
found, j
 
 Be aware that if the Dockerfile is found and the docker command works, test-patch will always
fail the build if the Dockerfile itself fails the build.  It will not attempt to continue
in the non-Docker mode.
 
-NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX) or C:\Users
(Windows) as the base and patchprocess directories (specified by the --basedir and --patch-dir
options respectively), because automatically mountable directories are limited to them. See
[the Docker documentation](https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume).
-
 Dockerfile images will be named with a test-patch prefix and suffix with either a date or
a git commit hash. By using this information, test-patch will automatically manage broken/stale
container images that are hanging around if it is run in --robot mode.  In this way, if Docker
fails to build the image, the disk space should eventually be cleaned and returned back to
the system.  The docker mode can also be run in a "safe" mode that prevents deletions via
the `--dockerdelrep` option.  Specifying this option will cause test-patch to only report
what it would have deleted, but not actually remove anything.
 
-Docker's `--memory` flag is supported via the `--dockermemlimit` option.  This enables the
container's memory size to be limited.  This may be important to set to prevent things like
broken unit tests bringing down the entire build server.  See  [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/)
for more details.
+## Resource Controls
+
+Docker's `--memory` flag is supported via the `--dockermemlimit` option.  This enables the
container's memory size to be limited.  This may be important to set to prevent things like
broken unit tests bringing down the entire build server.  See [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/)
for more details. Apache Yetus also sets the --oom-score-adj to 500 in order to offer itself
as the first processes to be killed if memory is low.
+
+Additionally, if bash v4 and Linux is in use, a separate process is launched to keep a rolling
count of the maximum number of threads (not processes!) in use at one time. This number will
be reported at the end of the test-patch run.  Depending upon the build, languages, features
enabled, etc, this number may be helpful in determining what the value of `--proclimit`
+
+# Process Reaper
+
+A common problem is the 'stuck' unit test. If bash v4.0 or higher is in use, Apache Yetus
may be told to turn on the process reaper functionality.  Using the `--reapearmode` option,
this feature may be configured to either report and even kill left over processes that match
provided regular expressions.
+
+  WARNING: Using `--reapermode` outside of Docker will report or kill ALL matching processes
on the system.  It is recommended to only use those options whilst in Docker mode.
 
-Additionally, Apache Yetus sets the --oom-score-adj to 500 in order to offer itself as the
first processes to be killed if memory is low.
+The reaper will run after every 'external' command that is printed on the console.  This
includes almost all build tool commands and individual test commands.
 
 # Plug-ins
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/asf-site-src/source/documentation/in-progress/precommit-basic.md
----------------------------------------------------------------------
diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md
index 1a82beb..90afc43 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-basic.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md
@@ -52,7 +52,7 @@ For Solaris and Solaris-like operating systems, the default location for
the POS
 test-patch requires these installed components to execute:
 
 * git-based project (and git 1.7.3 or higher installed)
-* bash v3.2 or higher
+* bash v3.2 or higher (bash v4.0 or higher is recommended)
 * GNU diff
 * GNU patch
 * POSIX awk
@@ -164,7 +164,7 @@ We used two new options here.  --basedir sets the location of the repository
to
 
 # Fork Bomb Protection
 
-By default, test-patch.sh will set the user soft limit (ulimit -Su) to a relatively low 1,000
processes (and, on some operating systems, threads!). This is to prevent errant processes
from eating up all system resources.  If this limit is too low (e.g., highly threaded Java
processes), it may be necessary to use the `--proclimit` option.  For example:
+By default, test-patch.sh will set the user soft limit (ulimit -Su) to a relatively low 1,000
processes (and, on some operating systems with some languages such as Java, threads!). This
is to prevent errant processes from eating up all system resources.  If this limit is too
low, it may be necessary to use the `--proclimit` option.  For example:
 
 ```bash
 $ test-patch --proclimit=10000
@@ -172,6 +172,8 @@ $ test-patch --proclimit=10000
 
 ... will set it to be 10,000 processes.
 
+  NOTE: The actual implementation of this feature is dependent upon the version of Bash.
 For bash v4 and higher (most operating systems), the fork bomb protection is generally only
used for the build and QA tools.  This means Apache Yetus should continue to function. For
earlier versions of bash (e.g., OS X), the limit is applied to all of test-patch. If the limit
is hit, Apache Yetus will itself likely crash.
+
 # Automation
 
 After the tests have run, there is a directory that contains all of the test-patch related
artifacts.  This is generally referred to as the patchprocess directory.  By default, test-patch
tries to make something off of /tmp to contain this content.  Using the `--patch-dir` option,
one can specify exactly which directory to use.  This is helpful for automated precommit testing
so that Jenkins or other automated workflow system knows where to look to gather up the output.
@@ -320,8 +322,8 @@ test-patch also has a mode to utilize Docker:
 $ test-patch.sh (other options) --docker
 ```
 
-This will do some preliminary setup and then re-execute itself inside a Docker container.
 For more information on how to provide a custom Dockerfile, see the advanced guide.
+This will do some preliminary setup and then re-execute itself inside a Docker container.
 For more information on how to provide a custom Dockerfile and other Docker-specific features,
see the advanced guide.
 
-## In Closing
+# In Closing
 
 test-patch has many other features and command line options for the basic user.  Many of
these are self-explanatory.  To see the list of options, run test-patch.sh without any options
or with --help.

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/precommit/core.d/01-common.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/01-common.sh b/precommit/core.d/01-common.sh
index d5057d7..e8dee09 100755
--- a/precommit/core.d/01-common.sh
+++ b/precommit/core.d/01-common.sh
@@ -569,3 +569,47 @@ function faster_dirname
     echo .
   fi
 }
+
+## @description  Set the USER_NAME, USER_ID, and GROUP_ID env vars
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function determine_user
+{
+  # On the Apache Jenkins hosts, $USER is pretty much untrustable because
+  # something sets it to an account that doesn't actually exist.
+  # Instead, we need to try and override it with something that's
+  # probably close to reality.
+  if [[ ${TESTPATCHMODE} =~ jenkins ]]; then
+    USER=$(id | cut -f2 -d\( | cut -f1 -d\))
+  fi
+
+  USER_NAME=${SUDO_USER:=$USER}
+  # shellcheck disable=SC2034
+  USER_ID=$(id -u "${USER_NAME}")
+  # shellcheck disable=SC2034
+  GROUP_ID=$(id -g "${USER_NAME}")
+}
+
+## @description  Kill a process id
+## @audience     private
+## @stability    evolving
+## @replaceable  yes
+## @param        pid
+function pid_kill
+{
+  declare pid=$1
+  declare cmd
+  declare kill_timeout=3
+
+  kill "${pid}" >/dev/null 2>&1
+  sleep "${kill_timeout}"
+  if kill -0 "${pid}" > /dev/null 2>&1; then
+    yetus_error "WARNING: ${pid} did not stop gracefully after ${kill_timeout} second(s):
Trying to kill with kill -9"
+    kill -9 "${pid}" >/dev/null 2>&1
+  fi
+  if ps -p "${pid}" > /dev/null 2>&1; then
+    cmd=$(ps -o args "${pid}")
+    yetus_error "ERROR: Unable to kill ${pid}: ${cmd}"
+  fi
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/precommit/core.d/docker.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/docker.sh b/precommit/core.d/docker.sh
index 48cba0a..6b06892 100755
--- a/precommit/core.d/docker.sh
+++ b/precommit/core.d/docker.sh
@@ -474,33 +474,6 @@ function docker_cleanup
   docker_container_maintenance
 }
 
-## @description  Deterine the user name and user id of the user
-## @description  that the docker container should use
-## @audience     private
-## @stability    evolving
-## @replaceable  no
-## @param        args
-function docker_determine_user
-{
-  # On the Apache Jenkins hosts, $USER is pretty much untrustable beacuse some
-  # ... person ... sets it to an account that doesn't actually exist.
-  # so instead, we need to try and override it with something that's
-  # probably close to reality.
-  if [[ ${TESTPATCHMODE} =~ jenkins ]]; then
-    USER=$(id | cut -f2 -d\( | cut -f1 -d\))
-  fi
-
-  if [[ "$(uname -s)" == "Linux" ]]; then
-    USER_NAME=${SUDO_USER:=$USER}
-    USER_ID=$(id -u "${USER_NAME}")
-    GROUP_ID=$(id -g "${USER_NAME}")
-  else # boot2docker uid and gid
-    USER_NAME=${USER}
-    USER_ID=1000
-    GROUP_ID=50
-  fi
-}
-
 ## @description  Determine the revision of a dockerfile
 ## @audience     private
 ## @stability    evolving
@@ -673,6 +646,6 @@ function docker_handler
   PATCH_DIR=$(relative_dir "${PATCH_DIR}")
 
   docker_cleanup
-  docker_determine_user
+  determine_user
   docker_run_image
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/precommit/core.d/reaper.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/reaper.sh b/precommit/core.d/reaper.sh
new file mode 100755
index 0000000..8a5f1dc
--- /dev/null
+++ b/precommit/core.d/reaper.sh
@@ -0,0 +1,252 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+REAPER_MODE=off   # off, report, kill
+declare -i REAPER_TOTAL_COUNT=0
+REAPER_DOCKER_ONLY=true
+REAPER_ZOMBIE_MODULES=()
+REAPER_ZOMBIE_LOGS=()
+declare -a REAPER_NAMES
+
+
+## @description  Add a regex to the reaper's checklist
+## @description  NOTE: Users WILL override anything added before
+## @description  argument parsing!
+## @stability    evolving
+## @audience     public
+## @replaceable  no
+function reaper_add_name
+{
+  yetus_add_array_element REAPER_NAMES "$1"
+}
+
+## @description  Reaper-specific usage
+## @stability    stable
+## @audience     private
+## @replaceable  no
+function reaper_usage
+{
+  yetus_add_option "--reapermode={off,report,kill}" "Set unit test reaper mode (default:
'${REAPER_MODE}')"
+  yetus_add_option "--reaperdockeronly=<bool>" "Only run the reaper in --docker (default:
${REAPER_DOCKER_ONLY})"
+  yetus_add_option "--reapernames=<list>" "List of regexs to search (default build
tool dependent)"
+}
+
+## @description  Reaper-specific argument parsing
+## @stability    stable
+## @audience     private
+## @replaceable  no
+## @params       arguments
+function reaper_parse_args
+{
+  declare i
+
+  for i in "$@"; do
+    case ${i} in
+      --reapermode=*)
+        REAPER_MODE=${i#*=}
+      ;;
+      --reaperdockeronly=*)
+        REAPER_DOCKER_ONLY=${i#*=}
+      ;;
+      --reapernames=*)
+        yetus_comma_to_array REAPER_NAMES "${i#*=}"
+      ;;
+    esac
+  done
+
+  # Running the reaper outside of Dockermode is very dangerous
+
+  if [[ "${REAPER_DOCKER_ONLY}" = "true" && ${DOCKERMODE} = "false" ]]; then
+    REAPER_MODE="off"
+    return
+  fi
+
+  # make sure REAPER_MODE is something valid and turn us on
+  # as necessary
+  if [[ "${REAPER_MODE}" = "report" || "${REAPER_MODE}" = "kill" ]]; then
+    add_test_format reaper
+    yetus_add_entry EXEC_MODES Reaper
+  else
+    REAPER_MODE="off"
+  fi
+
+}
+
+## @description  Initialize the reaper
+## @stability    stable
+## @audience     private
+## @replaceable  yes
+## @params       arguments
+function reaper_initialize
+{
+  determine_user
+}
+
+
+
+## @description  Reaper coprocessor function that
+## @description  runs outside the law
+## @stability    evolving
+## @audience     private
+## @replaceable  yes
+function reaper_coproc_func
+{
+  declare line
+  declare i
+  declare module
+  declare filefrag
+  declare cmd
+  declare args
+  declare pid
+  declare -a pidlist
+  declare -i count
+
+  echo "Reaper watching for: ${REAPER_NAMES[*]}" >> "${PATCH_DIR}/reaper.txt"
+
+  while true; do
+    read -r cmd
+    case ${cmd} in
+      reap)
+
+        read -r module
+        read -r logfile
+
+        while read -r line; do
+          ((count=count+1))
+          for i in "${REAPER_NAMES[@]}"; do
+            echo "${line}" | ${GREP} -E "${i}" >> "${PATCH_DIR}/${logfile}"
+          done
+        done < <(ps -u "${USER_ID}" -o pid= -o args=)
+
+        pidlist=()
+        count=0
+        while read -r line; do
+          ((count=count+1))
+          pid=$(echo "${line}" | cut -f1 -d' ')
+          args=$(echo "${line}" | cut -f2- -d' ')
+          if [[ "${REAPER_MODE}" = "kill" ]]; then
+            pidlist+=("${pid}")
+            echo "Killing ${pid} ${args}" >> "${PATCH_DIR}/reaper.txt" 2>&1
+          fi
+        done < <(cat "${PATCH_DIR}/${logfile}")
+
+        # tell our parent how many
+        # doing this now means killing in the background
+        echo ${count}
+
+        if [[ ${count} -eq 0 ]]; then
+          rm "${PATCH_DIR}/${logfile}"
+        fi
+
+        for i in "${pidlist[@]}"; do
+          if [[ "${REAPER_MODE}" = "kill" ]]; then
+            pid_kill "${i}" >> "${PATCH_DIR}/reaper.txt" 2>&1
+          fi
+        done
+      ;;
+      exit)
+        exit 0
+      ;;
+    esac
+  done
+}
+
+## @description  Run the reaper
+## @stability    evolving
+## @audience     private
+## @replaceable  yes
+## @params       module
+## @params       testlog
+## @params       testfrag
+function reaper_post_exec
+{
+  declare module=$1
+  declare filefrag=$2
+  declare count
+  declare myfile="${filefrag}-reaper.txt"
+  declare killmsg=""
+
+  case "${REAPER_MODE}" in
+    off)
+      return 0
+    ;;
+    kill)
+      killmsg=" and killed"
+    ;;
+  esac
+
+  yetus_debug "Checking for unreaped processes:"
+
+  # give some time for things to die naturally
+  sleep 2
+
+  #shellcheck disable=SC2154,SC2086
+  printf "reap\n%s\n%s\n" "${module}" "${myfile}" >&${reaper_coproc[1]}
+
+  #shellcheck disable=SC2154,SC2086
+  read -r count <&${reaper_coproc[0]}
+
+  if [[ ${count} -gt 0 ]]; then
+    ((REAPER_TOTAL_COUNT=REAPER_TOTAL_COUNT+count))
+    printf "\nFound%s %s left over processes\n\n" "${killmsg}" "${count}"
+    REAPER_ZOMBIE_MODULES+=("${module}:${count}")
+    REAPER_ZOMBIE_LOGS+=("@@BASE@@/${myfile}")
+    return 1
+  fi
+
+  return 0
+}
+
+## @description  Reaper output to the user
+## @stability    evolving
+## @audience     private
+## @replaceable  yes
+## @params       jdkname
+function reaper_finalize_results
+{
+  declare jdk=$1
+  declare fn
+
+  if [[ "${REAPER_MODE}" = "off" ]]; then
+    return 0
+  fi
+
+  if [[ ${#REAPER_ZOMBIE_MODULES[@]} -gt 0 ]] ; then
+    populate_test_table "${jdk}Unreaped Processes" "${REAPER_ZOMBIE_MODULES[@]}"
+    for fn in "${REAPER_ZOMBIE_LOGS[@]}"; do
+      add_footer_table "Unreaped Processes Log" "${fn}"
+    done
+    REAPER_ZOMBIE_MODULES=()
+    REAPER_ZOMBIE_LOGS=()
+  fi
+}
+
+## @description  Reaper output to the user
+## @stability    evolving
+## @audience     private
+## @replaceable  yes
+## @params       jdkname
+function reaper_total_count
+{
+
+  if [[ "${REAPER_MODE}" = "off" ]]; then
+    return 0
+  fi
+
+  if [[ ${REAPER_TOTAL_COUNT} -gt 0 ]]; then
+    add_vote_table -0 reaper "Unreaped process count: ${REAPER_TOTAL_COUNT}"
+  fi
+}

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/precommit/test-patch.d/maven.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/maven.sh b/precommit/test-patch.d/maven.sh
index e3865be..8f414b4 100755
--- a/precommit/test-patch.d/maven.sh
+++ b/precommit/test-patch.d/maven.sh
@@ -99,6 +99,9 @@ function maven_initialize
   maven_add_install mvnsite
   maven_add_install unit
 
+  # Tell the reaper about the maven surefire plugin
+  reaper_add_name surefirebooter
+
   # we need to do this before docker does it as root
   if [[ ! ${MAVEN_CUSTOM_REPOS_DIR} =~ ^/ ]]; then
     yetus_error "ERROR: --mvn-custom-repos-dir must be an absolute path."

http://git-wip-us.apache.org/repos/asf/yetus/blob/1dcbbeb3/precommit/test-patch.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.sh b/precommit/test-patch.sh
index 8b1a8cc..b507101 100755
--- a/precommit/test-patch.sh
+++ b/precommit/test-patch.sh
@@ -337,6 +337,13 @@ function prepopulate_footer
 ## @replaceable  no
 function finish_footer_table
 {
+  declare counter
+
+  if [[ -f "${PATCH_DIR}/threadcounter.txt" ]]; then
+    counter=$(cat "${PATCH_DIR}/threadcounter.txt")
+    add_footer_table "Max. process+thread count" "${counter} (vs. ulimit of ${PROC_LIMIT})"
+  fi
+
   add_footer_table "modules" "C: ${CHANGED_MODULES[*]} U: ${CHANGED_UNION_MODULES}"
 }
 
@@ -616,6 +623,23 @@ function compute_unidiff
   rm "${tmpfile}"
 }
 
+## @description helper function for echo_and_redirect
+## @audience private
+## @stability evolving
+## @replaceable no
+function e_a_r_helper
+{
+  declare logfile=$1
+  shift
+  declare params=("${@}")
+
+  # shellcheck disable=SC2034
+  coproc yrr_coproc {
+    ulimit -Su "${PROC_LIMIT}"
+    yetus_run_and_redirect "${logfile}" "${params[@]}"
+  }
+}
+
 ## @description  Print the command to be executing to the screen. Then
 ## @description  run the command, sending stdout and stderr to the given filename
 ## @description  This will also ensure that any directories in ${BASEDIR} have
@@ -629,7 +653,7 @@ function compute_unidiff
 ## @return       $?
 function echo_and_redirect
 {
-  local logfile=$1
+  declare logfile=$1
   shift
 
   verify_patchdir_still_exists
@@ -638,7 +662,32 @@ function echo_and_redirect
   # to the screen
   echo "cd $(pwd)"
   echo "${*} > ${logfile} 2>&1"
-  yetus_run_and_redirect "${logfile}" "${@}"
+
+  if [[ ${BASH_VERSINFO[0]} -gt 3 ]]; then
+
+    # use a coprocessor with the
+    # lower proc limit so that yetus can
+    # do stuff unimpacted by it
+
+    e_a_r_helper "${logfile}" "${@}" >> "${COPROC_LOGFILE}" 2>&1
+
+    # now that it's off as a separate process, we need to wait
+    # for it to finish. wait will either return 0, exit code
+    # of the coproc, or 127. all of which is
+    # perfectly fine for us.
+
+
+    # shellcheck disable=SC2154,SC2086
+    wait ${yrr_coproc_PID}
+
+  else
+
+    # if bash < 4 (e.g., OS X), just run it
+    # the ulimit was set earlier
+
+    yetus_run_and_redirect "${logfile}" "${params[@]}"
+  fi
+
 }
 
 ## @description is a given directory relative to BASEDIR?
@@ -766,6 +815,12 @@ function yetus_usage
   yetus_generic_columnprinter "${YETUS_OPTION_USAGE[@]}"
   yetus_reset_usage
 
+  echo ""
+  echo "Reaper options:"
+  reaper_usage
+  yetus_generic_columnprinter "${YETUS_OPTION_USAGE[@]}"
+  yetus_reset_usage
+
   for plugin in ${BUILDTOOLS} ${TESTTYPES} ${BUGSYSTEMS} ${TESTFORMATS}; do
     if declare -f ${plugin}_usage >/dev/null 2>&1; then
       echo ""
@@ -934,6 +989,8 @@ function parse_args
 
   docker_parse_args "$@"
 
+  reaper_parse_args "$@"
+
   if [[ -z "${PATCH_OR_ISSUE}"
        && "${BUILDMODE}" = patch ]]; then
     yetus_usage
@@ -995,6 +1052,7 @@ function parse_args
     fi
   fi
   PATCH_DIR=$(yetus_abs "${PATCH_DIR}")
+  COPROC_LOGFILE="${PATCH_DIR}/coprocessors.txt"
 
   # we need absolute dir for ${CONSOLE_REPORT_FILE}
   if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then
@@ -1985,6 +2043,7 @@ function modules_workers
   declare statusjdk
   declare result=0
   declare argv
+  declare execvalue
 
   if [[ "${BUILDMODE}" = full ]]; then
     repo="the source"
@@ -2030,8 +2089,12 @@ function modules_workers
       $("${BUILDTOOL}_executor" "${testtype}") \
       ${MODULEEXTRAPARAM[${modindex}]//@@@MODULEFN@@@/${fn}} \
       "${argv[@]}"
+    execvalue=$?
 
-    if [[ $? == 0 ]] ; then
+    reaper_post_exec "${modulesuffix}" "${repostatus}-${testtype}-${fn}"
+    ((execvalue = execvalue + $? ))
+
+    if [[ ${execvalue} == 0 ]] ; then
       module_status \
         ${modindex} \
         +1 \
@@ -2984,6 +3047,77 @@ function distclean
   return 0
 }
 
+## @description  Start any coprocessors
+## @audience     private
+## @stability    evolving
+## @replaceable  yes
+function start_coprocessors
+{
+  # Eventually, we might open this up for plugins
+  # and other operating environments
+  # but for now, this is private and only for us
+
+  if [[ "${BASH_VERSINFO[0]}" -gt 3 ]]; then
+
+    determine_user
+
+    if [[ "${OSTYPE}" = Linux && "${DOCKERMODE}" = true ]]; then
+
+      # this is really only even remotely close to
+      # accurate under Docker, for the time being.
+
+      # shellcheck disable=SC2034
+      coproc process_counter_coproc {
+        declare threadcount
+        declare maxthreadcount
+        declare cmd
+
+        sleep 2
+        while true; do
+          threadcount=$(ps -L -u "${USER_ID}" -o lwp 2>/dev/null | wc -l)
+          if [[ ${threadcount} -gt ${maxthreadcount} ]]; then
+            maxthreadcount="${threadcount}"
+            echo "${maxthreadcount}" > "${PATCH_DIR}/threadcounter.txt"
+          fi
+          read -r -t 2 cmd
+          case "${cmd}" in
+            exit)
+              exit 0
+            ;;
+          esac
+        done
+      }
+    fi
+
+    if [[ "${REAPER_MODE}" != "off" ]]; then
+      # shellcheck disable=SC2034
+      coproc reaper_coproc {
+        reaper_coproc_func
+      }
+    fi
+
+  fi
+}
+
+## @description  Stop any coprocessors
+## @audience     private
+## @stability    evolving
+## @replaceable  yes
+function stop_coprocessors
+{
+  # shellcheck disable=SC2154
+  if [[ -n "${process_counter_coproc_PID}" ]]; then
+    # shellcheck disable=SC2086
+    echo exit >&${process_counter_coproc[1]}
+  fi
+
+  #shellcheck disable=SC2154
+  if [[ -n "${reaper_coproc_PID}" ]]; then
+    # shellcheck disable=SC2086
+    echo exit >&${reaper_coproc[1]}
+  fi
+}
+
 ## @description  Setup to execute
 ## @audience     public
 ## @stability    evolving
@@ -3127,9 +3261,22 @@ else
   initialize "$@"
 fi
 
-ulimit -Su "${PROC_LIMIT}"
 
-yetus_debug "Changed process/Java native thread limit to ${PROC_LIMIT}"
+if [[ ${BASH_VERSINFO[0]} -gt 3 ]]; then
+  yetus_debug "Starting coprocessors"
+
+  # we need to catch out and err bz the coproc
+  # command is extremely noisy on both startup
+  # and shutdown
+  start_coprocessors >> "${COPROC_LOGFILE}" 2>&1
+else
+
+  # If we aren't using bash4 (e.g. OS X), then set the ulimit now.
+  # bash4 gets it set in an (on demand) coprocessor
+
+  ulimit -Su "${PROC_LIMIT}"
+  yetus_debug "Changed process/Java native thread limit to ${PROC_LIMIT}"
+fi
 
 add_vote_table H "Prechecks"
 
@@ -3157,13 +3304,14 @@ else
 
 fi
 
-
 compile_cycle patch
 
 add_vote_table H "Other Tests"
 
 runtests
 
+stop_coprocessors
+
 finish_vote_table
 
 finish_footer_table


Mime
View raw message