yetus-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sek...@apache.org
Subject [12/20] yetus git commit: YETUS-156. [Umbrella] build driver
Date Wed, 11 May 2016 21:41:34 GMT
YETUS-156. [Umbrella] build driver

This is a rebased and squashed commit of the following issues:

    YETUS-347. change docker handler to use arrays
    YETUS-348. add ability to write console report to a file
    YETUS-349. core engine changes for build driver
    YETUS-350. plug-in changes for build driver
    YETUS-353. runtime wrapper for build driver
    YETUS-351. documentation changes for build driver
    YETUS-352. hadoop and hbase personality changes for build driver
    YETUS-354. findbugs in build driver has problems
    YETUS-371. build driver: speed up checkstyle
    YETUS-357. build driver for hadoop shouldn't union unit tests
    YETUS-372. build driver: unit tests still mentions patches
    YETUS-373. build driver: maven install should be smarter
    YETUS-356. build driver: author plug-in still mentions patches
    YETUS-356. build driver: author plug-in still mentions patches (addendum)
    YETUS-374. need a way to ignore results from a list of tests
    YETUS-375. build driver: maven install actually needs to be dumber
    YETUS-376. add ability to write report to an HTML formatted file
    YETUS-384. In build driver mode, some plugins have misleading status text
    YETUS-385. test plugin for author tag detection should have a way to opt-out of specific instances
    YETUS-378. flink, hbase, jmeter personalities custom tests need to be made qbt aware
    YETUS-386. document --html-report-file and clean up htmlout's comments

Signed-off-by: Sean Busbey <busbey@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/yetus/repo
Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/612a0f2d
Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/612a0f2d
Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/612a0f2d

Branch: refs/heads/YETUS-379
Commit: 612a0f2d12af3c6713b04c29ba6406f17b29f0fa
Parents: e3ea8a4
Author: Allen Wittenauer <aw@apache.org>
Authored: Tue Mar 29 13:17:19 2016 -0700
Committer: Allen Wittenauer <aw@apache.org>
Committed: Mon Apr 25 08:00:09 2016 -0700

----------------------------------------------------------------------
 .../in-progress/precommit-advanced.md           |   5 +
 .../in-progress/precommit-buildtools.md         |   2 +-
 .../documentation/in-progress/precommit-qbt.md  |  64 +++++
 build.sh                                        |   3 +-
 precommit/core.d/00-yetuslib.sh                 |   2 +-
 precommit/core.d/01-common.sh                   |  22 ++
 precommit/core.d/builtin-bugsystem.sh           |  10 +
 precommit/core.d/docker.sh                      |  24 +-
 precommit/personality/flink.sh                  |  61 +++--
 precommit/personality/hadoop.sh                 |  14 +-
 precommit/personality/hbase.sh                  | 167 +++++++++++--
 precommit/personality/jmeter.sh                 |   3 +
 .../test-patch-docker/launch-test-patch.sh      |  29 ++-
 precommit/test-patch.d/ant.sh                   |   2 +-
 precommit/test-patch.d/asflicense.sh            |   8 +-
 precommit/test-patch.d/author.sh                | 118 ++++++++--
 precommit/test-patch.d/autoconf.sh              |  14 +-
 precommit/test-patch.d/checkstyle.sh            |  46 ++--
 precommit/test-patch.d/cmake.sh                 |   4 +-
 precommit/test-patch.d/findbugs.sh              |  29 ++-
 precommit/test-patch.d/gradle.sh                |   6 +-
 precommit/test-patch.d/htmlout.sh               | 229 ++++++++++++++++++
 precommit/test-patch.d/maven.sh                 |  43 +++-
 precommit/test-patch.d/perlcritic.sh            |  12 +-
 precommit/test-patch.d/pylint.sh                |  16 +-
 precommit/test-patch.d/rubocop.sh               |  12 +-
 precommit/test-patch.d/ruby-lint.sh             |  12 +-
 precommit/test-patch.d/shellcheck.sh            |   8 +-
 precommit/test-patch.d/shelldocs.sh             |   8 +-
 precommit/test-patch.d/test4tests.sh            |   4 +
 precommit/test-patch.d/whitespace.sh            |  38 ++-
 precommit/test-patch.d/xml.sh                   |   6 +-
 precommit/test-patch.sh                         | 232 +++++++++++++------
 33 files changed, 983 insertions(+), 270 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-advanced.md
----------------------------------------------------------------------
diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
index d64392a..953b44e 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
@@ -80,6 +80,9 @@ Similarly, there are other functions that may be defined during the test-patch r
 * pluginname\_initialize
     - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
 
+* pluginname\_docker\_support
+    - Perform any necessary setup to configure Docker support for the given plugin.  Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array.
+
 * pluginname\_precheck
     - executed prior to the patch being applied but after the git repository is setup.  Returning a fail status here will exit test-patch.
 
@@ -253,6 +256,8 @@ There are a handful of extremely important system variables that make life easie
 
 * CHANGED\_MODULES[@] is an array of all modules that house all of the CHANGED\_FILES[@].  Be aware that the root of the source tree is reported as '.'.
 
+* DOCKER\_EXTRAARGS[@] is an array of command line arguments to apply to the `docker run` command.
+
 * GITHUB\_REPO is to help test-patch when talking to Github.  If test-patch is given just a number on the command line, it will default to using this repo to determine the pull request.
 
 * JIRA\_ISSUE\_RE is to help test-patch when talking to JIRA.  It helps determine if the given project is appropriate for the given JIRA issue.

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
----------------------------------------------------------------------
diff --git a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
index e800135..bf530ba 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
@@ -102,7 +102,7 @@ For example, the gradle build tool does not have a standard way to execute check
 
 * pluginname\_docker\_support
 
-    - If this build tool requires extra settings on the `docker run` command line, this function should be defined and write those options into a file called `${PATCH_DIR}/buildtool-docker-params.txt`.  This is particularly useful for things like mounting volumes for repository caches.
+    - If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches.
 
        **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions.  It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command.
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-qbt.md
----------------------------------------------------------------------
diff --git a/asf-site-src/source/documentation/in-progress/precommit-qbt.md b/asf-site-src/source/documentation/in-progress/precommit-qbt.md
new file mode 100644
index 0000000..3e8f578
--- /dev/null
+++ b/asf-site-src/source/documentation/in-progress/precommit-qbt.md
@@ -0,0 +1,64 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+qbt
+===
+
+`qbt` is a command to execute test-patch without a patch.  It uses
+the same plug-ins and the same options as test-patch.  The only
+difference is that no patch file, location, etc should be supplied.
+It is meant to be a way to easily get test-patch's output on your
+current source tree.  It is suitable to be run as a regularly
+scheduled build as part of your overall development strategy.
+
+When using an automation tool, it may be useful to use the
+`--console-report-file` option to send the summary email to a
+file. This can then be used with systems like Jenkin's
+email-ext plug-in to send the output as an emailed report:
+
+```
+${FILE,path="<report-file-path>"}
+```
+
+For something a bit more structured, there is also the `--html-report-file`
+option.  Using this output, again with Jenkins' email-ext plug-in, it is
+possible to build some very nice looking output that is easily customized:
+
+```
+<html>
+<head>
+<style>
+table {
+    border-collapse: collapse;
+}
+table, th, td {
+   border: 1px solid black;
+}
+tr:nth-child(even){background-color: #f2f2f2}
+</style>
+</head>
+<body>
+<p>See the <a href="${BUILD_URL}">Jenkins Build</a> for more information.</p>
+<p>${CHANGES, format="<div>[%d] (%a) %m</div>"}</p>
+<p></p>
+${FILE,path="<report-file-path>"}
+</body></html>
+```
+
+NOTE: Be aware that ASF mailing lists do not allow HTML formatted email.

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/build.sh
----------------------------------------------------------------------
diff --git a/build.sh b/build.sh
index 9fecbec..d5f3904 100755
--- a/build.sh
+++ b/build.sh
@@ -169,6 +169,7 @@ cp -r shelldocs "${bin_tarball}/lib/"
 cp -r release-doc-maker "${bin_tarball}/lib/"
 
 cp -r precommit "${bin_tarball}/lib/"
+ln -s test-patch.sh "${bin_tarball}/lib/precommit/qbt.sh"
 
 mkdir -p "${bin_tarball}/bin"
 
@@ -204,7 +205,7 @@ exec "\$(dirname -- "\${BASH_SOURCE-0}")/../lib/release-doc-maker/releasedocmake
 EOF
 chmod +x "${bin_tarball}/bin/releasedocmaker"
 
-for utility in shelldocs/shelldocs.py \
+for utility in shelldocs/shelldocs.py precommit/qbt.sh \
                precommit/smart-apply-patch.sh precommit/test-patch.sh
 do
   wrapper=${utility##*/}

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/00-yetuslib.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/00-yetuslib.sh b/precommit/core.d/00-yetuslib.sh
index 90ffa16..25118cb 100755
--- a/precommit/core.d/00-yetuslib.sh
+++ b/precommit/core.d/00-yetuslib.sh
@@ -101,7 +101,7 @@ function yetus_run_and_redirect
 ## @description  Given a filename or dir, return the absolute version of it
 ## @audience     public
 ## @stability    stable
-## @param        directory
+## @param        fsobj
 ## @replaceable  no
 ## @return       0 success
 ## @return       1 failure

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/01-common.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/01-common.sh b/precommit/core.d/01-common.sh
index 0661f6e..f31d613 100755
--- a/precommit/core.d/01-common.sh
+++ b/precommit/core.d/01-common.sh
@@ -536,6 +536,9 @@ function verify_command
     yetus_error "executable for '${cmd_name}' was not specified."
     return 1
   fi
+  if [[ ! "${cmd_path}" =~ / ]]; then
+    cmd_path=$(command -v "${cmd_path}")
+  fi
   if [[ ! -f ${cmd_path} ]]; then
     yetus_error "executable '${cmd_path}' for '${cmd_name}' does not exist."
     return 1
@@ -546,3 +549,22 @@ function verify_command
   fi
   return 0
 }
+
+## @description  Faster dirname, given the assumption that
+## @description  dirs are always absolute (e.g., start with /)
+## @description  DO NOT USE with relative paths or where
+## @description  assumption may not be valid!
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+## @param        fileobj
+function faster_dirname
+{
+  declare o=$1
+
+  if [[ "${o}" =~ / ]]; then
+    echo "${o%/*}"
+  else
+    echo .
+  fi
+}

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/builtin-bugsystem.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/builtin-bugsystem.sh b/precommit/core.d/builtin-bugsystem.sh
index f83e7c9..c7d9a6c 100755
--- a/precommit/core.d/builtin-bugsystem.sh
+++ b/precommit/core.d/builtin-bugsystem.sh
@@ -45,6 +45,11 @@ function console_finalreport
   declare seccoladj=0
   declare spcfx=${PATCH_DIR}/spcl.txt
 
+  if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then
+    exec 6>&1
+    exec >"${CONSOLE_REPORT_FILE}"
+  fi
+
   if [[ ${result} == 0 ]]; then
     if [[ ${ROBOT} == false ]]; then
       if declare -f ${PROJECT_NAME}_console_success >/dev/null; then
@@ -154,4 +159,9 @@ function console_finalreport
     printf "%s\n" "${comment}"
     ((i=i+1))
   done
+
+  if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then
+    exec 1>&6 6>&-
+    cat "${CONSOLE_REPORT_FILE}"
+  fi
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/docker.sh
----------------------------------------------------------------------
diff --git a/precommit/core.d/docker.sh b/precommit/core.d/docker.sh
index e0706a3..062c8c1 100755
--- a/precommit/core.d/docker.sh
+++ b/precommit/core.d/docker.sh
@@ -23,6 +23,8 @@ DOCKERFAIL="fallback,continue,fail"
 DOCKERSUPPORT=false
 DOCKER_ENABLE_PRIVILEGED=true
 
+declare -a DOCKER_EXTRAARGS
+
 ####
 #### IMPORTANT
 ####
@@ -531,7 +533,7 @@ function docker_run_image
     cleanup_and_exit 1
   fi
 
-  big_console_header "Building patch image: ${patchimagename}"
+  big_console_header "Building ${BUILDMODE} image: ${patchimagename}"
   start_clock
   # using the base image, make one that is patch specific
   dockercmd build \
@@ -564,14 +566,13 @@ PatchSpecificDocker
     cleanup_and_exit 1
   fi
 
-  if [[ -f "${PATCH_DIR}/buildtool-docker-params.txt" ]]; then
-    extraargs=$(cat "${PATCH_DIR}/buildtool-docker-params.txt")
-  else
-    extraargs=""
+  if [[ "${DOCKER_ENABLE_PRIVILEGED}" = true ]]; then
+    DOCKER_EXTRAARGS=("--privileged" "${DOCKER_EXTRAARGS[@]}")
   fi
 
-  if [[ "${DOCKER_ENABLE_PRIVILEGED}" = true ]]; then
-    extraargs="${extraargs} --privileged "
+  if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then
+    touch "${CONSOLE_REPORT_FILE}"
+    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${CONSOLE_REPORT_FILE}:/testptch/console.txt")
   fi
 
   client=$(docker_version Client)
@@ -579,9 +580,8 @@ PatchSpecificDocker
 
   dockerversion="Client=${client} Server=${server}"
   if [[ ${PATCH_DIR} =~ ^/ ]]; then
-    # shellcheck disable=SC2086
     exec "${DOCKERCMD}" run --rm=true -i \
-      ${extraargs} \
+      "${DOCKER_EXTRAARGS[@]}" \
       -v "${PWD}:/testptch/${PROJECT_NAME}" \
       -v "${PATCH_DIR}:/testptch/patchprocess" \
       -u "${USER_NAME}" \
@@ -596,9 +596,8 @@ PatchSpecificDocker
       --name "${containername}" \
       "${patchimagename}"
   else
-    # shellcheck disable=SC2086
     exec "${DOCKERCMD}" run --rm=true -i \
-      ${extraargs} \
+      "${DOCKER_EXTRAARGS[@]}" \
       -v "${PWD}:/testptch/${PROJECT_NAME}" \
       -u "${USER_NAME}" \
       -w "/testptch/${PROJECT_NAME}" \
@@ -612,6 +611,9 @@ PatchSpecificDocker
       --name "${containername}" \
       "${patchimagename}"
   fi
+
+  # this should never get reached, but we put it here just in case
+  cleanup_and_exit 1
 }
 
 ## @description  Switch over to a Docker container

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/flink.sh
----------------------------------------------------------------------
diff --git a/precommit/personality/flink.sh b/precommit/personality/flink.sh
index 6b4f0a8..76fa418 100755
--- a/precommit/personality/flink.sh
+++ b/precommit/personality/flink.sh
@@ -31,6 +31,10 @@ function personality_globals
 
 add_test_type flinklib
 
+## @description  flinklib file filter
+## @audience     private
+## @stability    evolving
+## @param        filename
 function flinklib_filefilter
 {
   local filename=$1
@@ -42,6 +46,9 @@ function flinklib_filefilter
   fi
 }
 
+## @description  count files in the lib dir
+## @audience     private
+## @stability    evolving
 function flinklib_count
 {
   find "${BASEDIR}" \
@@ -50,42 +57,46 @@ function flinklib_count
     | wc -l
 }
 
-function flinklib_preapply
+## @description  check fliblib
+## @audience     private
+## @stability    evolving
+## @param        repostatus
+function flinklib_rebuild
 {
+  declare repostatus=$1
+
   start_clock
   big_console_header "${PATCH_BRANCH} flink library dependencies"
 
   if ! verify_needed_test flinklib; then
-    echo "Patch does not need flinklib testing."
+    echo "${BUILDMODEMSG} does not need flinklib testing."
     return 0
   fi
 
   pushd "${BASEDIR}" >/dev/null
-  echo_and_redirect "${PATCH_DIR}/branch-flinklib-root.txt" \
+  echo_and_redirect "${PATCH_DIR}/${repostatus}-flinklib-root.txt" \
      "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
   if [[ $? != 0 ]]; then
      add_vote_table -1 flinklib "Unable to determine flink libs in ${PATCH_BRANCH}."
   fi
-  FLINK_PRE_LIB_FILES=$(flinklib_count)
-  popd >/dev/null
-}
 
-function flinklib_postapply
-{
-  start_clock
-  big_console_header "Patch flink library dependencies"
-
-  if ! verify_needed_test flinklib; then
-    echo "Patch does not need flinklib testing."
+  if [[ ${repostatus} = branch ]]; then
+    FLINK_PRE_LIB_FILES=$(flinklib_count)
     return 0
+  else
+    FLINK_POST_LIB_FILES=$(flinklib_count)
   fi
-
-  pushd "${BASEDIR}" >/dev/null
-  echo_and_redirect "${PATCH_DIR}/patch-flinklib-root.txt" \
-     "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
-  FLINK_POST_LIB_FILES=$(flinklib_count)
   popd >/dev/null
 
+  if [[ "${BUILDMODE}" = full ]]; then
+    if [[ ${FLINK_POST_LIB_FILES} -gt 0 ]]; then
+      add_vote_table -1 flinklib "Lib folder dependencies are currently ${FLINK_POST_LIB_FILES}"
+      return 1
+    else
+      add_vote_table +1 flinklib "No lib folder dependencies!"
+      return 0
+    fi
+  fi
 
   if [[ "${FLINK_POST_LIB_FILES}" -gt "${FLINK_PRE_LIB_FILES}" ]]; then
     add_vote_table -1 flinklib "Patch increases lib folder dependencies from " \
@@ -99,16 +110,4 @@ function flinklib_postapply
       "$((FLINK_PRE_LIB_FILES-FLINK_POST_LIB_FILES))."
   fi
   return 0
-}
-
-function flinklib_rebuild
-{
-  declare repostatus=$1
-
-  if [[ "${repostatus}" = branch ]]; then
-    flinklib_preapply
-  else
-    flinklib_postinstall
-  fi
-}
-
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/hadoop.sh
----------------------------------------------------------------------
diff --git a/precommit/personality/hadoop.sh b/precommit/personality/hadoop.sh
index 59dfd2c..5b96994 100755
--- a/precommit/personality/hadoop.sh
+++ b/precommit/personality/hadoop.sh
@@ -46,8 +46,12 @@ function hadoop_order
 
   if [[ ${ordering} = normal ]]; then
     hadoopm="${CHANGED_MODULES[*]}"
-  elif  [[ ${ordering} = union ]]; then
+  elif [[ ${ordering} = union ]]; then
     hadoopm="${CHANGED_UNION_MODULES}"
+  elif [[ ${ordering} = mvnsrc ]]; then
+    hadoopm="${MAVEN_SRC_MODULES[*]}"
+  elif [[ ${ordering} = mvnsrctest ]]; then
+    hadoopm="${MAVEN_SRCTEST_MODULES[*]}"
   else
     hadoopm="${ordering}"
   fi
@@ -205,7 +209,7 @@ function personality_modules
         ordering=.
       fi
 
-      if [[ ${repostatus} = patch ]]; then
+      if [[ "${repostatus}" = patch && "${BUILDMODE}" = patch ]]; then
         echo "javadoc pre-reqs:"
         for i in hadoop-project \
           hadoop-common-project/hadoop-annotations; do
@@ -226,7 +230,7 @@ function personality_modules
     ;;
     mvninstall)
       extra="-DskipTests"
-      if [[ ${repostatus} = branch ]]; then
+      if [[ "${repostatus}" = branch || "${BUILDMODE}" = full ]]; then
         ordering=.
       fi
     ;;
@@ -236,7 +240,9 @@ function personality_modules
       fi
     ;;
     unit)
-      if [[ "${CHANGED_MODULES[*]}" =~ \. ]]; then
+      if [[ "${BUILDMODE}" = full ]]; then
+        ordering=mvnsrc
+      elif [[ "${CHANGED_MODULES[*]}" =~ \. ]]; then
         ordering=.
       fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/hbase.sh
----------------------------------------------------------------------
diff --git a/precommit/personality/hbase.sh b/precommit/personality/hbase.sh
index 352d2e3..5e6f0dc 100755
--- a/precommit/personality/hbase.sh
+++ b/precommit/personality/hbase.sh
@@ -22,17 +22,23 @@ personality_plugins "all"
 function personality_globals
 {
   #shellcheck disable=SC2034
+  PROJECT_NAME=hbase
+  #shellcheck disable=SC2034
   PATCH_BRANCH_DEFAULT=master
   #shellcheck disable=SC2034
   JIRA_ISSUE_RE='^HBASE-[0-9]+$'
   #shellcheck disable=SC2034
   GITHUB_REPO="apache/hbase"
 
+  # TODO use PATCH_BRANCH to select hadoop versions to use.
   # All supported Hadoop versions that we want to test the compilation with
-  HBASE_HADOOP_VERSIONS="2.4.1 2.5.2 2.6.0"
+  HBASE_HADOOP_VERSIONS="2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1"
+
+  # TODO use PATCH_BRANCH to select jdk versions to use.
 
   # Override the maven options
   MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}"
+
 }
 
 ## @description  Queue up modules for this personality
@@ -53,16 +59,20 @@ function personality_modules
   extra="-DHBasePatchProcess"
 
   if [[ ${repostatus} == branch
-     && ${testtype} == mvninstall ]];then
-     personality_enqueue_module . ${extra}
-     return
-   fi
+     && ${testtype} == mvninstall ]] ||
+     [[ "${BUILDMODE}" == full ]];then
+    personality_enqueue_module . ${extra}
+    return
+  fi
 
   if [[ ${testtype} = findbugs ]]; then
     for module in "${CHANGED_MODULES[@]}"; do
-      # skip findbugs on hbase-shell
+      # skip findbugs on hbase-shell and hbase-it. hbase-it has nothing
+      # in src/main/java where findbugs goes to look
       if [[ ${module} == hbase-shell ]]; then
         continue
+      elif [[ ${module} == hbase-it ]]; then
+        continue
       else
         # shellcheck disable=SC2086
         personality_enqueue_module ${module} ${extra}
@@ -71,6 +81,16 @@ function personality_modules
     return
   fi
 
+  if [[ ${testtype} = unit ]]; then
+    extra="${extra} -PrunAllTests"
+
+    # Inject the jenkins build-id for our surefire invocations
+    # Used by zombie detection stuff, even though we're not including that yet.
+    if [ -n "${BUILD_ID}" ]; then
+      extra="${extra} -Dbuild.id=${BUILD_ID}"
+    fi
+  fi
+
   for module in "${CHANGED_MODULES[@]}"; do
     # shellcheck disable=SC2086
     personality_enqueue_module ${module} ${extra}
@@ -78,9 +98,19 @@ function personality_modules
 }
 
 ###################################################
+# Below here are our one-off tests specific to hbase.
+# TODO break them into individual files so it's easier to maintain them?
+
+# TODO line length check? could ignore all java files since checkstyle gets them.
+
+###################################################
 
 add_test_type hadoopcheck
 
+## @description  hadoopcheck file filter
+## @audience     private
+## @stability    evolving
+## @param        filename
 function hadoopcheck_filefilter
 {
   local filename=$1
@@ -90,6 +120,10 @@ function hadoopcheck_filefilter
   fi
 }
 
+## @description  hadoopcheck test
+## @audience     private
+## @stability    evolving
+## @param        repostatus
 function hadoopcheck_rebuild
 {
   local repostatus=$1
@@ -113,7 +147,7 @@ function hadoopcheck_rebuild
         -Dhadoop-two.version="${hadoopver}"
     count=$(${GREP} -c ERROR "${logfile}")
     if [[ ${count} -gt 0 ]]; then
-      add_vote_table -1 hadoopcheck "Patch causes ${count} errors with Hadoop v${hadoopver}."
+      add_vote_table -1 hadoopcheck "${BUILDMODEMSG} causes ${count} errors with Hadoop v${hadoopver}."
       ((result=result+1))
     fi
   done
@@ -122,14 +156,19 @@ function hadoopcheck_rebuild
     return 1
   fi
 
-  add_vote_table +1 hadoopcheck "Patch does not cause any errors with Hadoop ${HBASE_HADOOP_VERSIONS}."
+  add_vote_table +1 hadoopcheck "${BUILDMODEMSG} does not cause any errors with Hadoop ${HBASE_HADOOP_VERSIONS}."
   return 0
 }
 
 ######################################
 
+# TODO if we need the protoc check, we probably need to check building all the modules that rely on hbase-protocol
 add_test_type hbaseprotoc
 
+## @description  hbaseprotoc file filter
+## @audience     private
+## @stability    evolving
+## @param        filename
 function hbaseprotoc_filefilter
 {
   local filename=$1
@@ -139,33 +178,38 @@ function hbaseprotoc_filefilter
   fi
 }
 
+## @description  hadoopcheck test
+## @audience     private
+## @stability    evolving
+## @param        repostatus
 function hbaseprotoc_rebuild
 {
-  local i=0
-  local fn
-  local module
-  local logfile
-  local count
-  local result
+  declare repostatus=$1
+  declare i=0
+  declare fn
+  declare module
+  declare logfile
+  declare count
+  declare result
 
   if [[ "${repostatus}" = branch ]]; then
     return 0
   fi
 
-  if ! verify_needed_test hbaseprotoc; then
+  verify_needed_test hbaseprotoc
+  if [[ $? == 0 ]]; then
     return 0
   fi
 
-  big_console_header "Patch HBase protoc plugin"
+  big_console_header "HBase protoc plugin: ${BUILDMODE}"
 
   start_clock
 
-
   personality_modules patch hbaseprotoc
   modules_workers patch hbaseprotoc compile -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess
 
   # shellcheck disable=SC2153
-  until [[ $i -eq ${#MODULE[@]} ]]; do
+  until [[ $i -eq "${#MODULE[@]}" ]]; do
     if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then
       ((result=result+1))
       ((i=i+1))
@@ -196,6 +240,10 @@ function hbaseprotoc_rebuild
 
 add_test_type hbaseanti
 
+## @description  hbaseanti file filter
+## @audience     private
+## @stability    evolving
+## @param        filename
 function hbaseanti_filefilter
 {
   local filename=$1
@@ -205,13 +253,22 @@ function hbaseanti_filefilter
   fi
 }
 
+## @description  hbaseanti patch file check
+## @audience     private
+## @stability    evolving
+## @param        filename
 function hbaseanti_patchfile
 {
   local patchfile=$1
   local warnings
   local result
 
-  if ! verify_needed_test hbaseanti; then
+  if [[ "${BUILDMODE}" = full ]]; then
+    return 0
+  fi
+
+  verify_needed_test hbaseanti
+  if [[ $? == 0 ]]; then
     return 0
   fi
 
@@ -238,3 +295,75 @@ function hbaseanti_patchfile
   add_vote_table +1 hbaseanti "" "Patch does not have any anti-patterns."
   return 0
 }
+
+
+## @description  hbase custom mvnsite file filter.  See HBASE-15042
+## @audience     private
+## @stability    evolving
+## @param        filename
+function mvnsite_filefilter
+{
+  local filename=$1
+
+  if [[ ${BUILDTOOL} = maven ]]; then
+    if [[ ${filename} =~ src/main/site || ${filename} =~ src/main/asciidoc ]]; then
+      yetus_debug "tests/mvnsite: ${filename}"
+      add_test mvnsite
+    fi
+  fi
+}
+
+## This is named so that yetus will check us right after running tests.
+## Essentially, we check for normal failures and then we look for zombies.
+#function hbase_unit_logfilter
+#{
+#  declare testtype="unit"
+#  declare input=$1
+#  declare output=$2
+#  declare processes
+#  declare process_output
+#  declare zombies
+#  declare zombie_count=0
+#  declare zombie_process
+#
+#  yetus_debug "in hbase-specific unit logfilter."
+#
+#  # pass-through to whatever is counting actual failures
+#  if declare -f ${BUILDTOOL}_${testtype}_logfilter >/dev/null; then
+#    "${BUILDTOOL}_${testtype}_logfilter" "${input}" "${output}"
+#  elif declare -f ${testtype}_logfilter >/dev/null; then
+#    "${testtype}_logfilter" "${input}" "${output}"
+#  fi
+#
+#  start_clock
+#  if [ -n "${BUILD_ID}" ]; then
+#    yetus_debug "Checking for zombie test processes."
+#    processes=$(jps -v | "${GREP}" surefirebooter | "${GREP}" -e "hbase.build.id=${BUILD_ID}")
+#    if [ -n "${processes}" ] && [ "$(echo "${processes}" | wc -l)" -gt 0 ]; then
+#      yetus_warn "Found some suspicious process(es). Waiting a bit to see if they're just slow to stop."
+#      yetus_debug "${processes}"
+#      sleep 30
+#      #shellcheck disable=SC2016
+#      for pid in $(echo "${processes}"| ${AWK} '{print $1}'); do
+#        # Test our zombie still running (and that it still an hbase build item)
+#        process_output=$(ps -p "${pid}" | tail +2 | "${GREP}" -e "hbase.build.id=${BUILD_ID}")
+#        if [[ -n "${process_output}" ]]; then
+#          yetus_error "Zombie: ${process_output}"
+#          ((zombie_count = zombie_count + 1))
+#          zombie_process=$(jstack "${pid}" | "${GREP}" -e "\.Test" | "${GREP}" -e "\.java"| head -3)
+#          zombies="${zombies} ${zombie_process}"
+#        fi
+#      done
+#    fi
+#    if [ "${zombie_count}" -ne 0 ]; then
+#      add_vote_table -1 zombies "There are ${zombie_count} zombie test(s)"
+#      populate_test_table "zombie unit tests" "${zombies}"
+#    else
+#      yetus_info "Zombie check complete. All test runs exited normally."
+#      stop_clock
+#    fi
+#  else
+#    add_vote_table -0 zombies "There is no BUILD_ID env variable; can't check for zombies."
+#  fi
+#
+#}

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/jmeter.sh
----------------------------------------------------------------------
diff --git a/precommit/personality/jmeter.sh b/precommit/personality/jmeter.sh
index 194fc4f..b4e61b5 100755
--- a/precommit/personality/jmeter.sh
+++ b/precommit/personality/jmeter.sh
@@ -58,6 +58,9 @@ function jmeter_parse_args
   done
 }
 
+## @description  Download jmetere dependencies
+## @audience     private
+## @stability    evolving
 function jmeter_precheck
 {
   if [[ ${JMETER_DOWNLOAD_JARS} = true ]]; then

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch-docker/launch-test-patch.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch-docker/launch-test-patch.sh b/precommit/test-patch-docker/launch-test-patch.sh
index 7daa612..f15dbf8 100755
--- a/precommit/test-patch-docker/launch-test-patch.sh
+++ b/precommit/test-patch-docker/launch-test-patch.sh
@@ -14,6 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+declare -a OVERWRITEARGS
+
+OVERWRITEARGS=("--reexec")
+OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--dockermode")
+OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--basedir=${BASEDIR}")
+
 cd "${BASEDIR}" || exit 1
 
 if [[ -n ${JAVA_HOME}
@@ -25,6 +31,10 @@ fi
 if [[ -z ${JAVA_HOME} ]]; then
   JAVA_HOME=$(find /usr/lib/jvm/ -name "java-*" -type d | tail -1)
   export JAVA_HOME
+  if [[ -n "${JAVA_HOME}" ]]; then
+    OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--java-home=${JAVA_HOME}")
+    echo "Setting ${JAVA_HOME} as the JAVA_HOME."
+  fi
 fi
 
 # Avoid out of memory errors in builds
@@ -35,9 +45,9 @@ export MAVEN_OPTS
 TESTPATCHMODE=${TESTPATCHMODE/--docker }
 TESTPATCHMODE=${TESTPATCHMODE%--docker}
 
-
-cd "${BASEDIR}" || exit 1
 PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P)
+OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--patch-dir=${PATCH_DIR}")
+OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--user-plugins=${PATCH_DIR}/precommit/user-plugins")
 
 # if patch system is generic, then it's either a local
 # patch file or was in some other way not pulled from a bug
@@ -45,16 +55,15 @@ PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P)
 # test-patch where to find it.
 if [[ "${PATCH_SYSTEM}" = generic ]]; then
   cp -p "${PATCH_DIR}/patch" /testptch/extras/patch
-  patchfile="/testptch/extras/patch"
+  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "/testptch/extras/patch")
+fi
+
+if [[ -f /testptch/console.txt ]]; then
+  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--console-report-file=/testptch/console.txt")
 fi
 
 cd "${PATCH_DIR}/precommit/" || exit 1
 #shellcheck disable=SC2086
 "${PATCH_DIR}/precommit/test-patch.sh" \
-   --reexec \
-   --dockermode ${TESTPATCHMODE} \
-   --basedir="${BASEDIR}" \
-   --patch-dir="${PATCH_DIR}" \
-   --java-home="${JAVA_HOME}" \
-   --user-plugins="${PATCH_DIR}/precommit/user-plugins" \
-   ${patchfile}
+   ${TESTPATCHMODE} \
+  "${OVERWRITEARGS[@]}"

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/ant.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/ant.sh b/precommit/test-patch.d/ant.sh
index 64c4067..61e13c9 100755
--- a/precommit/test-patch.d/ant.sh
+++ b/precommit/test-patch.d/ant.sh
@@ -200,5 +200,5 @@ function ant_builtin_personality_file_tests
 
 function ant_docker_support
 {
-  echo "-v ${HOME}/.ivy2:${HOME}/.ivy2" > "${PATCH_DIR}/buildtool-docker-params.txt"
+  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS}" "-v" "${HOME}/.ivy2:${HOME}/.ivy2")
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/asflicense.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/asflicense.sh b/precommit/test-patch.d/asflicense.sh
index edf7a7f..593e4af 100755
--- a/precommit/test-patch.d/asflicense.sh
+++ b/precommit/test-patch.d/asflicense.sh
@@ -61,7 +61,7 @@ function asflicense_tests
   local numpatch
   local btfails=true
 
-  big_console_header "Determining number of patched ASF License errors"
+  big_console_header "Determining number of ASF License errors"
 
   start_clock
 
@@ -105,7 +105,7 @@ function asflicense_tests
   # RAT fails the build if there are license problems.
   # so let's take advantage of that a bit.
   if [[ $? == 0 && ${btfails} = true ]]; then
-    add_vote_table 1 asflicense "Patch does not generate ASF License warnings."
+    add_vote_table 1 asflicense "${BUILDMODEMSG} does not generate ASF License warnings."
     return 0
   fi
 
@@ -134,7 +134,7 @@ function asflicense_tests
   echo "There appear to be ${numpatch} ASF License warnings after applying the patch."
   if [[ -n ${numpatch}
      && ${numpatch} -gt 0 ]] ; then
-    add_vote_table -1 asflicense "Patch generated ${numpatch} ASF License warnings."
+    add_vote_table -1 asflicense "${BUILDMODEMSG} generated ${numpatch} ASF License warnings."
 
     echo "Lines that start with ????? in the ASF License "\
         "report indicate files that do not have an Apache license header:" \
@@ -146,7 +146,7 @@ function asflicense_tests
     add_footer_table asflicense "@@BASE@@/patch-asflicense-problems.txt"
     return 1
   fi
-  add_vote_table 1 asflicense "Patch does not generate ASF License warnings."
+  add_vote_table 1 asflicense "${BUILDMODEMSG} does not generate ASF License warnings."
   return 0
 }
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/author.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/author.sh b/precommit/test-patch.d/author.sh
index 4497d03..dccd504 100755
--- a/precommit/test-patch.d/author.sh
+++ b/precommit/test-patch.d/author.sh
@@ -16,21 +16,83 @@
 
 add_test_type author
 
-## @description  Check the current directory for @author tags
+## @description  author usage hook
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function author_usage
+{
+  yetus_add_option "--author-ignore-list=<list>" "list of filenames to ignore (full build mode only)"
+}
+
+## @description  author parse args hook
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function author_parse_args
+{
+  declare i
+
+  for i in "$@"; do
+    case ${i} in
+      --author-ignore-list=*)
+        yetus_comma_to_array AUTHOR_IGNORE_LIST "${i#*=}"
+      ;;
+    esac
+  done
+}
+
+## @description  helper function for @author tags check
 ## @audience     private
 ## @stability    evolving
 ## @replaceable  no
 ## @return       0 on success
 ## @return       1 on failure
+function author_generic
+{
+  declare authortags
+  declare i
+  declare msg
+
+  if [[ "${BUILDMODE}" = full ]]; then
+    msg="source tree"
+  else
+    msg="patch"
+  fi
+
+  # shellcheck disable=SC2016
+  authortags=$(wc -l "${PATCH_DIR}/author-tags.txt" | "${AWK}" '{print $1}')
+  echo "There appear to be ${authortags} @author tags in the ${msg}."
+  if [[ ${authortags} != 0 ]] ; then
+    add_vote_table -1 @author \
+      "${BUILDMODEMSG} appears to contain ${authortags} @author tags which the" \
+      " community has agreed to not allow in code contributions."
+    add_footer_table @author "@@BASE@@/author-tags.txt"
+    return 1
+  fi
+  add_vote_table +1 @author "${BUILDMODEMSG} does not contain any @author tags."
+  return 0
+}
+
+## @description  Check the current patchfile for @author tags
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+## @return       0 on success
+## @return       1 on failure
+## @param        patchfile
 function author_patchfile
 {
   declare patchfile=$1
-  declare authorTags
   # shellcheck disable=SC2155
   declare -r appname=$(basename "${BASH_SOURCE-$0}")
   declare i
 
-  big_console_header "Checking there are no @author tags in the patch."
+  if [[ "${BUILDMODE}" != patch ]]; then
+    return
+  fi
+
+  big_console_header "Checking for @author tags: ${BUILDMODE}"
 
   start_clock
 
@@ -43,16 +105,44 @@ function author_patchfile
   done
 
   ${GREP} -i -n '^[^-].*@author' "${patchfile}" >> "${PATCH_DIR}/author-tags.txt"
-  # shellcheck disable=SC2016
-  authorTags=$(wc -l "${PATCH_DIR}/author-tags.txt" | "${AWK}" '{print $1}')
-  echo "There appear to be ${authorTags} @author tags in the patch."
-  if [[ ${authorTags} != 0 ]] ; then
-    add_vote_table -1 @author \
-      "The patch appears to contain ${authorTags} @author tags which the" \
-      " community has agreed to not allow in code contributions."
-    add_footer_table @author "@@BASE@@/author-tags.txt"
-    return 1
+  author_generic
+}
+
+
+## @description  Check the current directory for @author tags
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+## @return       0 on success
+## @return       1 on failure
+function author_postcompile
+{
+  # shellcheck disable=SC2155
+  declare -r appname=$(basename "${BASH_SOURCE-$0}")
+  declare fn
+
+  if [[ "${BUILDMODE}" != full ]]; then
+    return
   fi
-  add_vote_table +1 @author "The patch does not contain any @author tags."
-  return 0
+
+  big_console_header "Checking for @author tags: ${BUILDMODE}"
+
+  start_clock
+
+  "${GIT}" grep -n -I --extended-regexp -i -e '^[^-].*@author' \
+    | ${GREP} -v "${appname}" \
+    >> "${PATCH_DIR}/author-tags-git.txt"
+
+  if [[ -z "${AUTHOR_IGNORE_LIST[0]}" ]]; then
+    cp -p "${PATCH_DIR}/author-tags-git.txt" "${PATCH_DIR}/author-tags.txt"
+  else
+    cp -p "${PATCH_DIR}/author-tags-git.txt" "${PATCH_DIR}/author-tags.1"
+    for fn in "${AUTHOR_IGNORE_LIST[@]}"; do
+      ${GREP} -v -E "^${fn}" "${PATCH_DIR}/author-tags.1" >> "${PATCH_DIR}/author-tags.2"
+      mv "${PATCH_DIR}/author-tags.2" "${PATCH_DIR}/author-tags.1"
+    done
+    mv "${PATCH_DIR}/author-tags.1" "${PATCH_DIR}/author-tags.txt"
+  fi
+
+  author_generic
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/autoconf.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/autoconf.sh b/precommit/test-patch.d/autoconf.sh
index 2937e72..933a8c0 100755
--- a/precommit/test-patch.d/autoconf.sh
+++ b/precommit/test-patch.d/autoconf.sh
@@ -91,9 +91,9 @@ function autoconf_precompile
   fi
 
   if [[ "${repostatus}" = branch ]]; then
-    big_console_header "Pre-patch ${PATCH_BRANCH} autoconf verification"
+    big_console_header "autoconf verification: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch autoconf verification"
+    big_console_header "autoconf verification: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" autoreconf
@@ -108,7 +108,7 @@ function autoconf_precompile
       # shellcheck disable=SC2153
       add_vote_table -1 autoreconf "${PATCH_BRANCH} unable to autoreconf"
     else
-      add_vote_table -1 autoreconf "${repostatus} unable to autoreconf"
+      add_vote_table -1 autoreconf "${BUILDMODEMSG} is unable to autoreconf"
     fi
     add_footer_table "autoreconf" "@@BASE@@/${repostatus}-autoconf-autoreconf"
     return 1
@@ -117,7 +117,7 @@ function autoconf_precompile
       # shellcheck disable=SC2153
       add_vote_table +1 autoreconf "${PATCH_BRANCH} autoreconf successful"
     else
-      add_vote_table +1 autoreconf "${repostatus} autoreconf successful"
+      add_vote_table +1 autoreconf "${BUILDMODEMSG} can autoreconf"
     fi
   fi
 
@@ -138,7 +138,7 @@ function autoconf_precompile
       # shellcheck disable=SC2153
       add_vote_table -1 configure "${PATCH_BRANCH} unable to configure"
     else
-      add_vote_table -1 configure "${repostatus} unable to configure"
+      add_vote_table -1 configure "${BUILDMODEMSG} is unable to configure"
     fi
     add_footer_table "configure" "@@BASE@@/${repostatus}-autoconf-configure"
     return 1
@@ -147,7 +147,7 @@ function autoconf_precompile
       # shellcheck disable=SC2153
       add_vote_table +1 configure "${PATCH_BRANCH} configure successful"
     else
-      add_vote_table +1 configure "${repostatus} configure successful"
+      add_vote_table +1 configure "${BUILDMODEMSG} can configure"
     fi
   fi
   return 0
@@ -201,4 +201,4 @@ function autoconf_builtin_personality_file_tests
   else
     make_builtin_personality_file_tests "${filename}"
   fi
-}
\ No newline at end of file
+}

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/checkstyle.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/checkstyle.sh b/precommit/test-patch.d/checkstyle.sh
index 4363e4d..f0b3e57 100755
--- a/precommit/test-patch.d/checkstyle.sh
+++ b/precommit/test-patch.d/checkstyle.sh
@@ -140,7 +140,6 @@ function checkstyle_runner
   declare savestop
   declare output
   declare logfile
-  declare repo
   declare modulesuffix
   declare cmd
   declare logline
@@ -151,12 +150,6 @@ function checkstyle_runner
   # first, let's clear out any previous run information
   modules_reset
 
-  if [[ ${repostatus} == branch ]]; then
-    repo=${PATCH_BRANCH}
-  else
-    repo="the patch"
-  fi
-
   # loop through the modules we've been given
   #shellcheck disable=SC2153
   until [[ $i -eq ${#MODULE[@]} ]]; do
@@ -200,14 +193,14 @@ function checkstyle_runner
       > "${tmp}"
 
     if [[ $? == 0 ]] ; then
-      module_status ${i} +1 "${logfile}" "${modulesuffix} in ${repo} passed checkstyle"
+      module_status ${i} +1 "${logfile}" "${BUILDMODEMSG} ${modulesuffix} passed checkstyle"
     else
-      module_status ${i} -1 "${logfile}" "${modulesuffix} in ${repo} failed checkstyle"
+      module_status ${i} -1 "${logfile}" "${BUILDMODEMSG} ${modulesuffix} failed checkstyle"
       ((result = result + 1))
     fi
 
     # if we have some output, we need to do more work:
-    if [[ -s ${tmp} ]]; then
+    if [[ -s ${tmp} && "${BUILDMODE}" = patch ]]; then
 
       # first, let's pull out all of the files that
       # we actually care about, esp since that run
@@ -219,6 +212,7 @@ function checkstyle_runner
         ${GREP} "${j}" "${tmp}" >> "${tmp}.1"
       done
 
+
       # now that we have just the files we care about,
       # let's unscrew it. You see...
 
@@ -248,7 +242,8 @@ function checkstyle_runner
       popd >/dev/null
       # later on, calcdiff will turn this into code(:column):error
       # compare, and then put the file:line back onto it.
-
+    else
+      cp -p "${tmp}" "${output}"
     fi
 
     rm "${tmp}" "${tmp}.1" 2>/dev/null
@@ -288,7 +283,7 @@ function checkstyle_preapply
     return 0
   fi
 
-  big_console_header "${PATCH_BRANCH} checkstyle"
+  big_console_header "checkstyle: ${PATCH_BRANCH}"
 
   start_clock
 
@@ -321,7 +316,7 @@ function checkstyle_postapply
     return 0
   fi
 
-  big_console_header "Patch checkstyle plugin"
+  big_console_header "checkstyle: ${BUILDMODE}"
 
   start_clock
 
@@ -346,12 +341,21 @@ function checkstyle_postapply
     module=${MODULE[$i]}
     fn=$(module_file_fragment "${module}")
 
-    # call calcdiffs to allow overrides
-    calcdiffs \
-      "${PATCH_DIR}/branch-checkstyle-${fn}.txt" \
-      "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \
-      checkstyle \
-      > "${PATCH_DIR}/diff-checkstyle-${fn}.txt"
+    # if there is no comparison to be done,
+    # we can speed this up tremendously
+    if [[ "${BUILDMODE}" = full ]]; then
+      touch  "${PATCH_DIR}/branch-checkstyle-${fn}.txt"
+      cp -p "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \
+        "${PATCH_DIR}/diff-checkstyle-${fn}.txt"
+    else
+
+      # call calcdiffs to allow overrides
+      calcdiffs \
+        "${PATCH_DIR}/branch-checkstyle-${fn}.txt" \
+        "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \
+        checkstyle \
+        > "${PATCH_DIR}/diff-checkstyle-${fn}.txt"
+    fi
 
     #shellcheck disable=SC2016
     numbranch=$(wc -l "${PATCH_DIR}/branch-checkstyle-${fn}.txt" | ${AWK} '{print $1}')
@@ -371,9 +375,9 @@ function checkstyle_postapply
 
     if [[ ${addpatch} -gt 0 ]] ; then
       ((result = result + 1))
-      module_status ${i} -1 "diff-checkstyle-${fn}.txt" "${mod}: patch ${statstring}"
+      module_status ${i} -1 "diff-checkstyle-${fn}.txt" "${mod}: ${BUILDMODEMSG} ${statstring}"
     elif [[ ${fixedpatch} -gt 0 ]]; then
-      module_status ${i} +1 "diff-checkstyle-${fn}.txt" "${mod}: patch ${statstring}"
+      module_status ${i} +1 "diff-checkstyle-${fn}.txt" "${mod}: ${BUILDMODEMSG} ${statstring}"
       summarize=false
     fi
     ((i=i+1))

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/cmake.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/cmake.sh b/precommit/test-patch.d/cmake.sh
index 9b5f265..7e68a4d 100755
--- a/precommit/test-patch.d/cmake.sh
+++ b/precommit/test-patch.d/cmake.sh
@@ -128,9 +128,9 @@ function cmake_precompile
 
   if [[ "${repostatus}" = branch ]]; then
     # shellcheck disable=SC2153
-    big_console_header "${PATCH_BRANCH} cmake CMakeLists.txt"
+    big_console_header "cmake CMakeLists.txt: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch cmake CMakeLists.txt"
+    big_console_header "cmake CMakeLists.txt: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" CMakeLists.txt

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/findbugs.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/findbugs.sh b/precommit/test-patch.d/findbugs.sh
index e4399fc..24ccc95 100755
--- a/precommit/test-patch.d/findbugs.sh
+++ b/precommit/test-patch.d/findbugs.sh
@@ -95,10 +95,8 @@ function findbugs_precheck
 ## @audience     private
 ## @stability    evolving
 ## @replaceable  no
-## @param        repostatus
 function findbugs_maven_skipper
 {
-  declare repostat=$1
   declare -i i=0
   declare skiplist=()
   declare modname
@@ -122,7 +120,11 @@ function findbugs_maven_skipper
   done
 
   if [[ -n "${modname}" ]]; then
-    add_vote_table 0 findbugs "Skipped ${repostat} modules with no Java source: ${skiplist[*]}"
+    if [[ "${BUILDMODE}" = patch ]]; then
+      add_vote_table 0 findbugs "Skipped patched modules with no Java source: ${skiplist[*]}"
+    else
+      add_vote_table 0 findbugs "Skipped ${#skiplist[@]} modules in the source tree with no Java source."
+    fi
   fi
 }
 
@@ -132,6 +134,7 @@ function findbugs_maven_skipper
 ## @replaceable  no
 ## @return       0 on success
 ## @return       1 on failure
+## @param        repostatus
 function findbugs_runner
 {
   local name=$1
@@ -147,7 +150,7 @@ function findbugs_runner
   # strip out any modules that aren't actually java modules
   # this can save a lot of time during testing
   if [[ "${BUILDTOOL}" = maven ]]; then
-    findbugs_maven_skipper "${name}"
+    findbugs_maven_skipper
   fi
 
   "${BUILDTOOL}_modules_worker" "${name}" findbugs
@@ -248,7 +251,7 @@ function findbugs_preapply
     return 0
   fi
 
-  big_console_header "Pre-patch findbugs detection"
+  big_console_header "findbugs detection: ${PATCH_BRANCH}"
 
   findbugs_runner branch
   result=$?
@@ -278,9 +281,19 @@ function findbugs_preapply
 
     if [[ ${module_findbugs_warnings} -gt 0 ]] ; then
       msg="${module} in ${PATCH_BRANCH} has ${module_findbugs_warnings} extant Findbugs warnings."
-      if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" ]]; then
+      if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" = "true" ]]; then
+        module_status ${modindex} -1 "branch-findbugs-${fn}-warnings.html" "${msg}"
+        ((result=result+1))
+      elif [[ "${BUILDMODE}" = full ]]; then
         module_status ${modindex} -1 "branch-findbugs-${fn}-warnings.html" "${msg}"
         ((result=result+1))
+        populate_test_table FindBugs "module:${module}"
+        #shellcheck disable=SC2162
+        while read line; do
+          firstpart=$(echo "${line}" | cut -f2 -d:)
+          secondpart=$(echo "${line}" | cut -f9- -d' ')
+          add_test_table "" "${firstpart}:${secondpart}"
+        done < <("${FINDBUGS_HOME}/bin/convertXmlToText" "${warnings_file}.xml")
       else
         module_status ${modindex} 0 "branch-findbugs-${fn}-warnings.html" "${msg}"
       fi
@@ -329,7 +342,7 @@ function findbugs_postinstall
     return 0
   fi
 
-  big_console_header "Patch findbugs detection"
+  big_console_header "findbugs detection: ${BUILDMODE}"
 
   findbugs_runner patch
 
@@ -463,7 +476,7 @@ function findbugs_rebuild
 {
   declare repostatus=$1
 
-  if [[ "${repostatus}" = branch ]]; then
+  if [[ "${repostatus}" = branch || "${BUILDMODE}" = full ]]; then
     findbugs_preapply
   else
     findbugs_postinstall

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/gradle.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/gradle.sh b/precommit/test-patch.d/gradle.sh
index 57e023d..e809465 100755
--- a/precommit/test-patch.d/gradle.sh
+++ b/precommit/test-patch.d/gradle.sh
@@ -106,9 +106,9 @@ function gradle_precompile
 
   if [[ "${repostatus}" = branch ]]; then
     # shellcheck disable=SC2153
-    big_console_header "${PATCH_BRANCH} gradle bootstrap"
+    big_console_header "gradle boostrap: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch gradle bootstrap"
+    big_console_header "gradle bootstrap: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" gradleboot
@@ -277,5 +277,5 @@ function gradle_builtin_personality_file_tests
 
 function gradle_docker_support
 {
-  echo "-v ${HOME}/.gradle:${HOME}/.gradle" > "${PATCH_DIR}/buildtool-docker-params.txt"
+  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.gradle:${HOME}/.gradle")
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/htmlout.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/htmlout.sh b/precommit/test-patch.d/htmlout.sh
new file mode 100755
index 0000000..bf6d06e
--- /dev/null
+++ b/precommit/test-patch.d/htmlout.sh
@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+add_bugsystem htmlout
+
+## @description  Usage info for htmlout plugin
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function htmlout_usage
+{
+  yetus_add_option "--html-report-file=<file>" "Save the final report to an HTML-formated file"
+}
+
+## @description  Option parsing for htmlout plugin
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function htmlout_parse_args
+{
+  declare i
+  declare fn
+
+  for i in "$@"; do
+    case ${i} in
+      --html-report-file=*)
+        fn=${i#*=}
+      ;;
+    esac
+  done
+
+  if [[ -n "${fn}" ]]; then
+    touch "${fn}" 2>/dev/null
+    if [[ $? != 0 ]]; then
+      yetus_error "WARNING: cannot create ${fn}. Ignoring."
+    else
+      HTMLOUT_REPORTFILE=$(yetus_abs "${fn}")
+    fi
+  fi
+}
+
+## @description  Give access to the HTML report file in docker mode
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+function htmlout_docker_support
+{
+  if [[ -n ${HTMLOUT_REPORTFILE} ]]; then
+    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HTMLOUT_REPORTFILE}:${HTMLOUT_REPORTFILE}")
+  fi
+}
+
+
+## @description  Write out an HTML version of the final report to a file
+## @audience     private
+## @stability    evolving
+## @replaceable  no
+## @param        runresult
+function htmlout_finalreport
+{
+  declare result=$1
+  declare i
+  declare commentfile="${HTMLOUT_REPORTFILE}"
+  declare comment
+  declare vote
+  declare ourstring
+  declare ela
+  declare subs
+  declare color
+  declare comment
+
+  rm "${commentfile}" 2>/dev/null
+
+  if [[ -z "${HTMLOUT_REPORTFILE}" ]]; then
+    return
+  fi
+
+  big_console_header "Writing HTML to ${commentfile}"
+
+  {
+    echo "<table><tbody>"
+
+    if [[ ${result} == 0 ]]; then
+      echo "<tr><th><font color=\"green\">+1 overall</font></th></tr>"
+    else
+      echo "<tr><th><font color=\"red\">-1 overall</font></th></tr>"
+    fi
+    echo "</table></tbody>"
+    echo "<p></p>"
+  } >  "${commentfile}"
+
+  i=0
+  until [[ $i -eq ${#TP_HEADER[@]} ]]; do
+    ourstring=$(echo "${TP_HEADER[${i}]}" | tr -s ' ')
+    comment=$(echo "${ourstring}"  | cut -f2 -d\|)
+    printf "<tr><td>%s</td></tr>\n" "${comment}"
+    ((i=i+1))
+  done
+
+  {
+    echo "<table><tbody>"
+    echo "<tr>"
+    echo "<th>Vote</th>"
+    echo "<th>Subsystem</th>"
+    echo "<th>Runtime</th>"
+    echo "<th>Comment</th>"
+    echo "</tr>"
+  } >> "${commentfile}"
+
+  i=0
+  until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do
+    ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ')
+    vote=$(echo "${ourstring}" | cut -f2 -d\| | tr -d ' ')
+    subs=$(echo "${ourstring}"  | cut -f3 -d\|)
+    ela=$(echo "${ourstring}" | cut -f4 -d\|)
+    comment=$(echo "${ourstring}"  | cut -f5 -d\|)
+
+    # summary line
+    if [[ -z ${vote}
+      && -n ${ela} ]]; then
+      color="black"
+    elif [[ -z ${vote} ]]; then
+      # keep same color
+      true
+    else
+      # new vote line
+      case ${vote} in
+        1|"+1")
+          color="green"
+        ;;
+        -1)
+          color="red"
+        ;;
+        0)
+          color="blue"
+        ;;
+        *)
+          color="black"
+        ;;
+      esac
+    fi
+
+    {
+      echo "<tr>"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${vote}"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${subs}"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${ela}"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${comment}"
+      echo "</tr>"
+    } >> "${commentfile}"
+    ((i=i+1))
+  done
+  {
+    echo "</tbody></table>"
+    echo "<p></p>"
+  } >> "${commentfile}"
+
+  if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then
+    {
+      echo "<table><tbody>"
+      echo "<tr>"
+      echo "<th>Reason</th>"
+      echo "<th>Tests</th>"
+      echo "</tr>"
+    } >> "${commentfile}"
+
+    i=0
+    until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do
+      ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ')
+      subs=$(echo "${ourstring}"  | cut -f2 -d\|)
+      comment=$(echo "${ourstring}"  | cut -f3 -d\|)
+      {
+        echo "<tr>"
+        printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${subs}"
+        printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${comment}"
+        echo "</tr>"
+      } >> "${commentfile}"
+      ((i=i+1))
+    done
+
+    {
+      echo "</tbody></table>"
+      echo "<p></p>"
+    } >> "${commentfile}"
+  fi
+
+  {
+    echo "<table><tbody>"
+    echo "<tr>"
+    echo "<th>Subsystem</th>"
+    echo "<th>Report/Notes</th>"
+    echo "</tr>"
+  } >> "${commentfile}"
+
+  i=0
+  until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do
+    ourstring=$(echo "${TP_FOOTER_TABLE[${i}]}" |
+              ${SED} -e "s,@@BASE@@,${BUILD_URL}${BUILD_URL_ARTIFACTS},g" |
+              tr -s ' ')
+    subs=$(echo "${ourstring}"  | cut -f2 -d\|)
+    comment=$(echo "${ourstring}"  | cut -f3 -d\|)
+    {
+      echo "<tr>"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${subs}"
+      printf "<td><font color=\"%s\">%s</font></td>" "${color}" "${comment}"
+      echo "</tr>"
+    } >> "${commentfile}"
+    ((i=i+1))
+  done
+  {
+    echo "</tbody></table>"
+    echo "<p></p>"
+  } >> "${commentfile}"
+
+  printf "<p>This message was automatically generated.</p>" >> "${commentfile}"
+}

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/maven.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/maven.sh b/precommit/test-patch.d/maven.sh
index 2e006c1..3d37c37 100755
--- a/precommit/test-patch.d/maven.sh
+++ b/precommit/test-patch.d/maven.sh
@@ -95,6 +95,11 @@ function maven_parse_args
 
 function maven_initialize
 {
+  if ! verify_command "maven" "${MAVEN}"; then
+    return 1
+  fi
+
+  # we need to do this before docker does it as root
 
   maven_add_install mvneclipse
   maven_add_install mvnsite
@@ -139,7 +144,7 @@ function maven_precheck
   fi
 
   if [[ ${MAVEN_CUSTOM_REPOS} = true ]]; then
-    MAVEN_LOCAL_REPO="${MAVEN_CUSTOM_REPOS_DIR}/${PROJECT_NAME}-${PATCH_BRANCH}-${INSTANCE}"
+    MAVEN_LOCAL_REPO="${MAVEN_CUSTOM_REPOS_DIR}/${PROJECT_NAME}-${PATCH_BRANCH}-${BUILDMODE}-${INSTANCE}"
     if [[ -e "${MAVEN_LOCAL_REPO}"
        && ! -d "${MAVEN_LOCAL_REPO}" ]]; then
       yetus_error "ERROR: ${MAVEN_LOCAL_REPO} is not a directory."
@@ -369,7 +374,8 @@ function maven_builtin_personality_modules
   # this always makes sure the local repo has a fresh
   # copy of everything per pom rules.
   if [[ ${repostatus} == branch
-        && ${testtype} == mvninstall ]];then
+        && ${testtype} == mvninstall ]] ||
+     [[ "${BUILDMODE}" = full ]];then
     personality_enqueue_module "${CHANGED_UNION_MODULES}"
     return
   fi
@@ -450,9 +456,9 @@ function mvnsite_postcompile
   fi
 
   if [[ "${repostatus}" = branch ]]; then
-    big_console_header "Pre-patch ${PATCH_BRANCH} maven site verification"
+    big_console_header "maven site verification: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch maven site verification"
+    big_console_header "maven site verification: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" mvnsite
@@ -485,9 +491,9 @@ function mvneclipse_postcompile
   fi
 
   if [[ "${repostatus}" = branch ]]; then
-    big_console_header "Pre-patch ${PATCH_BRANCH} maven eclipse verification"
+    big_console_header "maven eclipse verification: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch maven eclipse verification"
+    big_console_header "maven eclipse verification: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" mvneclipse
@@ -537,9 +543,9 @@ function maven_precompile
   fi
 
   if [[ "${repostatus}" = branch ]]; then
-    big_console_header "Pre-patch ${PATCH_BRANCH} maven install"
+    big_console_header "maven install: ${PATCH_BRANCH}"
   else
-    big_console_header "Patch maven install"
+    big_console_header "maven install: ${BUILDMODE}"
   fi
 
   personality_modules "${repostatus}" mvninstall
@@ -554,11 +560,10 @@ function maven_precompile
 
 function maven_docker_support
 {
-  echo "-v ${HOME}/.m2:${HOME}/.m2" > "${PATCH_DIR}/buildtool-docker-params.txt"
+  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.m2:${HOME}/.m2")
 
   if [[ ${MAVEN_CUSTOM_REPOS} = true ]]; then
-    echo "-v ${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}" \
-      >> "${PATCH_DIR}/buildtool-docker-params.txt"
+    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}")
   fi
 }
 
@@ -676,5 +681,19 @@ function maven_reorder_modules
   yetus_debug "Maven: finish re-ordering modules"
   yetus_debug "Finished list: ${CHANGED_MODULES[*]}"
 
-  add_vote_table 0 mvndep "Maven dependency ordering for ${repostatus}"
+  # build some utility module lists for maven modules
+  for index in "${CHANGED_MODULES[@]}"; do
+    if [[ -d "${index}/src" ]]; then
+      MAVEN_SRC_MODULES=("${MAVEN_SRC_MODULES[@]}" "${index}")
+      if [[ -d "${index}/src/test" ]]; then
+        MAVEN_SRCTEST_MODULES=("${MAVEN_SRCTEST_MODULES[@]}" "${index}")
+      fi
+    fi
+  done
+
+  if [[ "${BUILDMODE}" = patch ]]; then
+    add_vote_table 0 mvndep "Maven dependency ordering for ${repostatus}"
+  else
+    add_vote_table 0 mvndep "Maven dependency ordering"
+  fi
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/perlcritic.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/perlcritic.sh b/precommit/test-patch.d/perlcritic.sh
index 8297f46..291e94e 100755
--- a/precommit/test-patch.d/perlcritic.sh
+++ b/precommit/test-patch.d/perlcritic.sh
@@ -64,11 +64,11 @@ function perlcritic_preapply
     return 0
   fi
 
-  big_console_header "Perl::Critic plugin: prepatch"
+  big_console_header "Perl::Critic plugin: ${PATCH_BRANCH}"
 
   start_clock
 
-  echo "Running perlcritic against modified perl scripts/modules."
+  echo "Running perlcritic against identified perl scripts/modules."
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
     if [[ ${i} =~ \.p[lm]$ && -f ${i} ]]; then
@@ -106,7 +106,7 @@ function perlcritic_postapply
     return 0
   fi
 
-  big_console_header "Perl::Critic plugin: postpatch"
+  big_console_header "Perl::Critic plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -114,7 +114,7 @@ function perlcritic_postapply
   # by setting the clock back
   offset_clock "${PERLCRITIC_TIMER}"
 
-  echo "Running perlcritic against modified perl scripts/modules."
+  echo "Running perlcritic against identified perl scripts/modules."
   # we re-check this in case one has been added
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
@@ -147,11 +147,11 @@ function perlcritic_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]]; then
-    add_vote_table -1 perlcritic "The applied patch ${statstring}"
+    add_vote_table -1 perlcritic "${BUILDMODEMSG} ${statstring}"
     add_footer_table perlcritic "@@BASE@@/diff-patch-perlcritic.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 perlcritic "The applied patch ${statstring}"
+    add_vote_table +1 perlcritic "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/pylint.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/pylint.sh b/precommit/test-patch.d/pylint.sh
index 2a9e1f8..7decb96 100755
--- a/precommit/test-patch.d/pylint.sh
+++ b/precommit/test-patch.d/pylint.sh
@@ -71,11 +71,11 @@ function pylint_preapply
     return 0
   fi
 
-  big_console_header "pylint plugin: prepatch"
+  big_console_header "pylint plugin: ${PATCH_BRANCH}"
 
   start_clock
 
-  echo "Running pylint against modified python scripts."
+  echo "Running pylint against identified python scripts."
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
     if [[ ${i} =~ \.py$ && -f ${i} ]]; then
@@ -87,7 +87,7 @@ function pylint_preapply
   if [[ -f ${PATCH_DIR}/${pylintStderr} ]]; then
     count=$(${GREP} -vc "^No config file found" "${PATCH_DIR}/${pylintStderr}")
     if [[ ${count} -gt 0 ]]; then
-      add_footer_table pylint "prepatch stderr: @@BASE@@/${pylintStderr}"
+      add_footer_table pylint "${PATCH_BRANCH} stderr: @@BASE@@/${pylintStderr}"
       return 1
     fi
   fi
@@ -113,7 +113,7 @@ function pylint_postapply
     return 0
   fi
 
-  big_console_header "pylint plugin: postpatch"
+  big_console_header "pylint plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -121,7 +121,7 @@ function pylint_postapply
   # by setting the clock back
   offset_clock "${PYLINT_TIMER}"
 
-  echo "Running pylint against modified python scripts."
+  echo "Running pylint against identified python scripts."
   # we re-check this in case one has been added
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
@@ -135,7 +135,7 @@ function pylint_postapply
     count=$(${GREP} -vc "^No config file found" "${PATCH_DIR}/${pylintStderr}")
     if [[ ${count} -gt 0 ]]; then
       add_vote_table -1 pylint "Something bad seems to have happened in running pylint. Please check pylint stderr files."
-      add_footer_table pylint "postpatch stderr: @@BASE@@/${pylintStderr}"
+      add_footer_table pylint "${BUILDMODEMSG} stderr: @@BASE@@/${pylintStderr}"
       return 1
     fi
   fi
@@ -158,11 +158,11 @@ function pylint_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]] ; then
-    add_vote_table -1 pylint "The applied patch ${statstring}"
+    add_vote_table -1 pylint "${BUILDMODEMSG} ${statstring}"
     add_footer_table pylint "@@BASE@@/diff-patch-pylint.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 pylint "The applied patch ${statstring}"
+    add_vote_table +1 pylint "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/rubocop.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/rubocop.sh b/precommit/test-patch.d/rubocop.sh
index 8e1d5b8..828bea1 100755
--- a/precommit/test-patch.d/rubocop.sh
+++ b/precommit/test-patch.d/rubocop.sh
@@ -64,11 +64,11 @@ function rubocop_preapply
     return 0
   fi
 
-  big_console_header "rubocop plugin: prepatch"
+  big_console_header "rubocop plugin: ${PATCH_BRANCH}"
 
   start_clock
 
-  echo "Running rubocop against modified ruby scripts."
+  echo "Running rubocop against identified ruby scripts."
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
     if [[ ${i} =~ \.rb$ && -f ${i} ]]; then
@@ -106,7 +106,7 @@ function rubocop_postapply
     return 0
   fi
 
-  big_console_header "rubocop plugin: postpatch"
+  big_console_header "rubocop plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -114,7 +114,7 @@ function rubocop_postapply
   # by setting the clock back
   offset_clock "${RUBOCOP_TIMER}"
 
-  echo "Running rubocop against modified ruby scripts."
+  echo "Running rubocop against identified ruby scripts."
   # we re-check this in case one has been added
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
@@ -146,11 +146,11 @@ function rubocop_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]] ; then
-    add_vote_table -1 rubocop "The applied patch ${statstring}"
+    add_vote_table -1 rubocop "${BUILDMODEMSG} ${statstring}"
     add_footer_table rubocop "@@BASE@@/diff-patch-rubocop.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 rubocop "The applied patch ${statstring}"
+    add_vote_table +1 rubocop "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/ruby-lint.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/ruby-lint.sh b/precommit/test-patch.d/ruby-lint.sh
index 94d4f53..78be7b6 100755
--- a/precommit/test-patch.d/ruby-lint.sh
+++ b/precommit/test-patch.d/ruby-lint.sh
@@ -63,11 +63,11 @@ function ruby_lint_preapply
     return 0
   fi
 
-  big_console_header "ruby-lint plugin: prepatch"
+  big_console_header "ruby-lint plugin: ${PATCH_BRANCH}"
 
   start_clock
 
-  echo "Running ruby-lint against modified ruby scripts."
+  echo "Running ruby-lint against identified ruby scripts."
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
     if [[ ${i} =~ \.rb$ && -f ${i} ]]; then
@@ -136,7 +136,7 @@ function ruby_lint_postapply
     return 0
   fi
 
-  big_console_header "ruby-lint plugin: postpatch"
+  big_console_header "ruby-lint plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -144,7 +144,7 @@ function ruby_lint_postapply
   # by setting the clock back
   offset_clock "${RUBY_LINT_TIMER}"
 
-  echo "Running ruby-lint against modified ruby scripts."
+  echo "Running ruby-lint against identified ruby scripts."
   # we re-check this in case one has been added
   pushd "${BASEDIR}" >/dev/null
   for i in "${CHANGED_FILES[@]}"; do
@@ -176,11 +176,11 @@ function ruby_lint_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]] ; then
-    add_vote_table -1 ruby-lint "The applied patch ${statstring}"
+    add_vote_table -1 ruby-lint "${BUILDMODEMSG} ${statstring}"
     add_footer_table ruby-lint "@@BASE@@/diff-patch-ruby-lint.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 ruby-lint "The applied patch ${statstring}"
+    add_vote_table +1 ruby-lint "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/shellcheck.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/shellcheck.sh b/precommit/test-patch.d/shellcheck.sh
index e9a7e15..0e053e3 100755
--- a/precommit/test-patch.d/shellcheck.sh
+++ b/precommit/test-patch.d/shellcheck.sh
@@ -106,7 +106,7 @@ function shellcheck_preapply
     return 0
   fi
 
-  big_console_header "shellcheck plugin: prepatch"
+  big_console_header "shellcheck plugin: ${PATCH_BRANCH}"
 
   start_clock
 
@@ -157,7 +157,7 @@ function shellcheck_postapply
     return 0
   fi
 
-  big_console_header "shellcheck plugin: postpatch"
+  big_console_header "shellcheck plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -194,12 +194,12 @@ function shellcheck_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]] ; then
-    add_vote_table -1 shellcheck "The applied patch ${statstring}"
+    add_vote_table -1 shellcheck "${BUILDMODEMSG} ${statstring}"
     add_footer_table shellcheck "@@BASE@@/diff-patch-shellcheck.txt"
     bugsystem_linecomments "shellcheck" "${PATCH_DIR}/diff-patch-shellcheck.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 shellcheck "The applied patch ${statstring}"
+    add_vote_table +1 shellcheck "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/shelldocs.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/shelldocs.sh b/precommit/test-patch.d/shelldocs.sh
index fea982a..d52fa6b 100755
--- a/precommit/test-patch.d/shelldocs.sh
+++ b/precommit/test-patch.d/shelldocs.sh
@@ -103,7 +103,7 @@ function shelldocs_preapply
     return 0
   fi
 
-  big_console_header "shelldocs plugin: prepatch"
+  big_console_header "shelldocs plugin: ${PATCH_BRANCH}"
 
   start_clock
 
@@ -134,7 +134,7 @@ function shelldocs_postapply
     return 0
   fi
 
-  big_console_header "shelldocs plugin: postpatch"
+  big_console_header "shelldocs plugin: ${BUILDMODE}"
 
   start_clock
 
@@ -170,12 +170,12 @@ function shelldocs_postapply
   statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
 
   if [[ ${diffPostpatch} -gt 0 ]] ; then
-    add_vote_table -1 shelldocs "The applied patch ${statstring}"
+    add_vote_table -1 shelldocs "${BUILDMODEMSG} ${statstring}"
     add_footer_table shelldocs "@@BASE@@/diff-patch-shelldocs.txt"
     bugsystem_linecomments "shelldocs" "${PATCH_DIR}/diff-patch-shelldocs.txt"
     return 1
   elif [[ ${fixedpatch} -gt 0 ]]; then
-    add_vote_table +1 shelldocs "The applied patch ${statstring}"
+    add_vote_table +1 shelldocs "${BUILDMODEMSG} ${statstring}"
     return 0
   fi
 

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/test4tests.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/test4tests.sh b/precommit/test-patch.d/test4tests.sh
index 644fbf9..6c3230c 100755
--- a/precommit/test-patch.d/test4tests.sh
+++ b/precommit/test-patch.d/test4tests.sh
@@ -27,6 +27,10 @@ function test4tests_patchfile
   declare testReferences=0
   declare i
 
+  if [[ "${BUILDMODE}" = full ]]; then
+    return
+  fi
+
   big_console_header "Checking there are new or changed tests in the patch."
 
   if ! verify_needed_test unit; then

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/whitespace.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/whitespace.sh b/precommit/test-patch.d/whitespace.sh
index a613d81..da0789b 100755
--- a/precommit/test-patch.d/whitespace.sh
+++ b/precommit/test-patch.d/whitespace.sh
@@ -49,21 +49,37 @@ function whitespace_postcompile
   start_clock
 
   pushd "${BASEDIR}" >/dev/null
-  # shellcheck disable=SC2016
-  ${AWK} '/\t/ {print $0}' \
-      "${GITDIFFCONTENT}" \
-    | ${GREP} -v Makefile: >> "${PATCH_DIR}/whitespace-tabs.txt"
 
-   ${GREP} -E '[[:blank:]]$' \
-     "${GITDIFFCONTENT}" \
-      >> "${PATCH_DIR}/whitespace-eol.txt"
+  case "${BUILDMODE}" in
+    patch)
+      # shellcheck disable=SC2016
+      ${AWK} '/\t/ {print $0}' \
+          "${GITDIFFCONTENT}" \
+        | ${GREP} -v Makefile: >> "${PATCH_DIR}/whitespace-tabs.txt"
+
+       ${GREP} -E '[[:blank:]]$' \
+         "${GITDIFFCONTENT}" \
+          >> "${PATCH_DIR}/whitespace-eol.txt"
+    ;;
+    full)
+      ${GIT} grep -n -I --extended-regexp '[[:blank:]]$' \
+         >> "${PATCH_DIR}/whitespace-eol.txt"
+      ${GIT} grep -n -I $'\t' \
+        | "${GREP}" -v Makefile \
+        >> "${PATCH_DIR}/whitespace-tabs.txt"
+    ;;
+  esac
 
   # shellcheck disable=SC2016
   count=$(wc -l "${PATCH_DIR}/whitespace-eol.txt" | ${AWK} '{print $1}')
 
   if [[ ${count} -gt 0 ]]; then
-    add_vote_table -1 whitespace "The patch has ${count}"\
-      " line(s) that end in whitespace. Use git apply --whitespace=fix."
+    if [[ "${BUILDMODE}" = full ]]; then
+      add_vote_table -1 whitespace "${BUILDMODEMSG} has ${count} line(s) that end in whitespace."
+    else
+      add_vote_table -1 whitespace \
+        "${BUILDMODEMSG} has ${count} line(s) that end in whitespace. Use git apply --whitespace=fix."
+    fi
 
     whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-eol.txt" "end of line"
     add_footer_table whitespace "@@BASE@@/whitespace-eol.txt"
@@ -74,7 +90,7 @@ function whitespace_postcompile
   count=$(wc -l "${PATCH_DIR}/whitespace-tabs.txt" | ${AWK} '{print $1}')
 
   if [[ ${count} -gt 0 ]]; then
-    add_vote_table -1 whitespace "The patch has ${count}"\
+    add_vote_table -1 whitespace "${BUILDMODEMSG} ${count}"\
       " line(s) with tabs."
     add_footer_table whitespace "@@BASE@@/whitespace-tabs.txt"
     whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-tabs.txt" "tabs in line"
@@ -87,6 +103,6 @@ function whitespace_postcompile
   fi
 
   popd >/dev/null
-  add_vote_table +1 whitespace "Patch has no whitespace issues."
+  add_vote_table +1 whitespace "${BUILDMODEMSG} has no whitespace issues."
   return 0
 }

http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/xml.sh
----------------------------------------------------------------------
diff --git a/precommit/test-patch.d/xml.sh b/precommit/test-patch.d/xml.sh
index 08e0a61..6460522 100755
--- a/precommit/test-patch.d/xml.sh
+++ b/precommit/test-patch.d/xml.sh
@@ -48,7 +48,7 @@ function xml_postcompile
     return 0
   fi
 
-  big_console_header "Checking if XML files are well-formed"
+  big_console_header "XML verification: ${BUILDMODE}"
 
   js="${JAVA_HOME}/bin/jrunscript"
 
@@ -65,13 +65,13 @@ function xml_postcompile
   done
 
   if [[ ${count} -gt 0 ]]; then
-    add_vote_table -1 xml "The patch has ${count} ill-formed XML file(s)."
+    add_vote_table -1 xml "${BUILDMODEMSG} has ${count} ill-formed XML file(s)."
     add_footer_table xml "@@BASE@@/xml.txt"
     popd >/dev/null
     return 1
   fi
 
   popd >/dev/null
-  add_vote_table +1 xml "The patch has no ill-formed XML file."
+  add_vote_table +1 xml "${BUILDMODEMSG} has no ill-formed XML file."
   return 0
 }


Mime
View raw message