qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject qpid-python-test script: should skipped tests be considered failures?
Date Mon, 27 Jan 2014 21:20:51 GMT
The qpid-python-test script has a facility for skipping tests (by
raising a Skipped exception) which works fine BUT if tests are skipped
the script exits with non-0 status - i.e. failure.

I propose we change this behavior. It is clear in the test output that
tests were skipped rather than failed, but that's not very useful when
the script is incorporated in larger test suites, CI frameworks etc.
where returning non-0 will be considered a failure and set people off in
failure investigation mode only to cause much annoyance when they dig
down and find skipped, not failed tests are causing the alarms. 

IMO skipping a test is different from failing - you skip because you
can't run the test for some environmental reason that has no bearing on
whether the functionality works, e.g. there's some library not installed
or what have you. We should certainly try to get skip stats reflected in
higher level tools that measure test health but I don't think we should
be ringing alarm bells as if something had failed.

Opinions? This behavior has been around for a long time so I'm wary of
changing it unilaterally. It is however the reason I've never used the
skip functionality and resorted to hacks like making tests pass but
print SKIP which is not as nice as doing it properly. 

To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

View raw message