mxnet-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Markham (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MXNET-1078) Installation instructions should be validated automatically
Date Tue, 16 Oct 2018 16:57:00 GMT

     [ https://issues.apache.org/jira/browse/MXNET-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Markham updated MXNET-1078:
---------------------------------
    Description: 
h2. Overview & Discussion

Some installation instructions can be tested automatically. However, when the instructions
are coupled with text for the user, updates can easily break automation scripts. Also, some
installation instructions follow a pattern of "if–>then" with multiple steps offering
multiple options. This creates a difficult situation for an automation process. 

For example, a contributor has worked on getting MXNet to work a Mac with one of those plugin
GPUs. They want to add new instructions for this. Most other contributors don't have access
to this setup and cannot verify the steps. We should not expect the contributor to update
the instructions, accommodate CI, and also fix the installation instructions nightly test. 

Another example would be that new users have no idea what math library they could or should
use, and probably don't care if even presented the option. Other more advanced users may have
preferences for what math library to use. Both sets of users find these options in the dependencies
sections, and the setup guide offers several different math library approaches. Which pathway
is more important to test? Can a script to test the installation instructions parse all the
options and test them all? 

 

I suggest the following split approach:
 # Simplify the installation instructions and follow a common template. Test the consolidated
install commands for the recommended build per platform per binding.
 # As secondary project to developer later, or in parallel if resources are available, use
the same format for the consolidated install commands to automate testing of the possible
variations of environment. 

h2. Install Templates
 * Create a template with a common flow for the installation instructions for any platform
or language binding.
 * The flow should consist of the steps the user should follow and code block containing each
command that should be executed. If there is more than one code block, the page should end
with a complete code block that has all of the steps in sequence; one that can be executed
directly if it were copied and pasted.
 * A nightly CI script can then be invoked to execute the consolidated installation code
block that the instructions use. 
 * PRs for installation instructions should be validated against these requirements.

Refer to this [Installation Guide Template|https://gist.github.com/aaronmarkham/132c5bf044bc179b9eb25e866b6ba102] gist
for an example using MXNet-R.
h2. QA Testing All Major Install Variants
 * For Mac, Ubuntu, Windows, and possibly CentOS test each build option and supported dependency
option.
 * This will lead to a large number of unique tests.
 * It is less ideal to use a CI system that is fully loaded with all dependencies because
it will miss issues when users try for a minimal installation. Therefore when testing a variation
it should be a clean install with only the candidate dependencies installed.

h2. Optional approaches / enhancements
 * The code blocks can be sourced by an include directive. This will pull the contents of
a text file that contains the commands.
 * Multiple CI scripts can be created - one for each platform and binding combination, so
it is clear what fails when everything else is passing. You can also disable a broken test
much easier, and not disable everything or have constant failures when one thing is broken.
 * Timing on implementation using includes is subject to getting the include feature to properly
work with Sphinx, or when Sphinx is replaced with Jekyll for the website builds. Both platforms
support includes, however the Sphinx implementation we use today seems partially broken and
doesn't render includes as described in the Sphinx documentation.

  was:
Some installation instructions can be tested automatically. However, when the instructions
are coupled with text for the user, updates can easily break automation scripts. Also, some
installation instructions follow a pattern of "if–>then" with multiple steps offering
multiple options. This creates a difficult situation for an automation process. 

For example, a contributor has worked on getting MXNet to work a Mac with one of those plugin
GPUs. They want to add new instructions for this. Most other contributors don't have access
to this setup and cannot verify the steps. We should not expect the contributor to update
the instructions, accommodate CI, and also fix the installation instructions nightly test. 

Another example would be that new users have no idea what math library they could or should
use, and probably don't care if even presented the option. Other more advanced users may have
preferences for what math library to use. Both sets of users find these options in the dependencies
sections, and the setup guide offers several different math library approaches. Which pathway
is more important to test? Can a script to test the installation instructions parse all the
options and test them all? 

 

I suggest the following approach.
 # Create a template with a common flow for the installation instructions for any platform
or language binding.
 # The flow should consist of the steps the user should follow and code block containing each
command that should be executed. If there is more than one code block, the page should end
with a complete code block that has all of the steps in sequence; one that can be executed
directly if it were copied and pasted.
 # A nightly CI script can then be invoked to execute the consolidated installation code
block that the instructions use. 
 # PRs for installation instructions should be validated against these requirements.

Optional approaches / enhancements
 # The code blocks can be sourced by an include directive. This will pull the contents of
a text file that contains the commands.
 # Alternatively, multiple CI scripts can be created - one for each platform and binding combination,
so it is clear what fails when everything else is passing. You can also disable a broken test
much easier, and not disable everything or have constant failures when one thing is broken.

 

Timing on implementation is subject to getting the include feature to properly work with Sphinx,
or when Sphinx is replaced with Jekyll for the website builds. Both platforms support includes,
however the Sphinx implementation we use today seems partially broken and doesn't render
includes as described in the Sphinx documentation.


> Installation instructions should be validated automatically
> -----------------------------------------------------------
>
>                 Key: MXNET-1078
>                 URL: https://issues.apache.org/jira/browse/MXNET-1078
>             Project: Apache MXNet
>          Issue Type: Story
>            Reporter: Aaron Markham
>            Priority: Major
>
> h2. Overview & Discussion
> Some installation instructions can be tested automatically. However, when the instructions
are coupled with text for the user, updates can easily break automation scripts. Also, some
installation instructions follow a pattern of "if–>then" with multiple steps offering
multiple options. This creates a difficult situation for an automation process. 
> For example, a contributor has worked on getting MXNet to work a Mac with one of those
plugin GPUs. They want to add new instructions for this. Most other contributors don't have
access to this setup and cannot verify the steps. We should not expect the contributor to
update the instructions, accommodate CI, and also fix the installation instructions nightly
test. 
> Another example would be that new users have no idea what math library they could or
should use, and probably don't care if even presented the option. Other more advanced users
may have preferences for what math library to use. Both sets of users find these options
in the dependencies sections, and the setup guide offers several different math library approaches.
Which pathway is more important to test? Can a script to test the installation instructions
parse all the options and test them all? 
>  
> I suggest the following split approach:
>  # Simplify the installation instructions and follow a common template. Test the consolidated
install commands for the recommended build per platform per binding.
>  # As secondary project to developer later, or in parallel if resources are available,
use the same format for the consolidated install commands to automate testing of the possible
variations of environment. 
> h2. Install Templates
>  * Create a template with a common flow for the installation instructions for any platform
or language binding.
>  * The flow should consist of the steps the user should follow and code block containing
each command that should be executed. If there is more than one code block, the page should
end with a complete code block that has all of the steps in sequence; one that can be executed
directly if it were copied and pasted.
>  * A nightly CI script can then be invoked to execute the consolidated installation
code block that the instructions use. 
>  * PRs for installation instructions should be validated against these requirements.
> Refer to this [Installation Guide Template|https://gist.github.com/aaronmarkham/132c5bf044bc179b9eb25e866b6ba102] gist
for an example using MXNet-R.
> h2. QA Testing All Major Install Variants
>  * For Mac, Ubuntu, Windows, and possibly CentOS test each build option and supported
dependency option.
>  * This will lead to a large number of unique tests.
>  * It is less ideal to use a CI system that is fully loaded with all dependencies because
it will miss issues when users try for a minimal installation. Therefore when testing a variation
it should be a clean install with only the candidate dependencies installed.
> h2. Optional approaches / enhancements
>  * The code blocks can be sourced by an include directive. This will pull the contents
of a text file that contains the commands.
>  * Multiple CI scripts can be created - one for each platform and binding combination,
so it is clear what fails when everything else is passing. You can also disable a broken test
much easier, and not disable everything or have constant failures when one thing is broken.
>  * Timing on implementation using includes is subject to getting the include feature
to properly work with Sphinx, or when Sphinx is replaced with Jekyll for the website builds.
Both platforms support includes, however the Sphinx implementation we use today seems partially
broken and doesn't render includes as described in the Sphinx documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


Mime
View raw message