james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Fondermann" <bernd.fonderm...@googlemail.com>
Subject Re: A failing or slow test is a good test too
Date Tue, 14 Nov 2006 08:31:33 GMT
On 11/13/06, Joachim Draeger <jdraeger@gmx.de> wrote:
> Am Montag, den 13.11.2006, 12:00 +0100 schrieb Bernd Fondermann:
>
> > > IMO a failing test is as valuable as a passing one. Maybe even more
> > > because it reminds us to do something.
> >
> > > I don't think that it is an indicator of quality to have always 100% of
> > > tests passing.
> >
> > My unit test 1x1 says: test runs have a binary result. It is either
> > green or red. If red, you have to fix the test. If you do nothing, the
> > test will soon be accompanied by a second failing test. Nobody checks
> > for failing tests anymore.
>
> I do not propose to do nothing when a test changes from green to red. I
> propose committing tests that fail because you don't have a solution so
> far.
> I can't see the advantage in accepting only passing tests.

Because thats the fundamental unit test paradigm. The whole red/green
thing is build on this.

If you know something fails, you write a test testing exactly this and
document it in code + in JIRA.

>
> > That does not necessarily mean every failing tests is a subject to
> > immediate fix. For example, this is not possible in
> > test-driven-development which is based on failing tests.
>
> Would you accept failing tests used for TDD in James?

As long as you are developing locally, yes.
Before the commit, all tests should pass. If work is resumed next day
and the feature gets implemented, the test magically gets to your
attention because it is now failing and gets inverted. No searching in
large sections of commented or uncommitted code. It is very
straightforward.

(Side note: Some unit test experts even say that new tests should be
written to fail once and changed to succeed afterwards. Only this way
it is proven that the test is able to detect a breakage of the thing
it is testing.)

>
> > But intended-to-fail failing tests obscures all
> > oops-something-bad-happened failing tests.
>
> I think that is only a management/administrative issue. If there are a
> lot of tests we need a workflow not to oversee any changes in the
> results.

And the only way IMHO to achieve this is to be green all (read: most)
of the time.

> Do you refer to psychological aspects? I consider the message "don't
> start working on a test, it could fail" more bad than "there is one
> failing test, nobody would mind a second...".

If somebody starts working on tests, they _will_ fail at the
beginning. But go away with a failing test and do no clean ups is not
polite.

Just like checking in non-compiling code. You could say, well it is
not compiling, never mind. I get back to that sometime.

>
> > > There often are known bugs for a long time, some of them not even
> > > documented anywhere.
> >
> > Agreed. It is a good thing to have tests that document
> > bugs/errors/missing features/evolving code.
> > But these test should not be failing, be commented out or be contained
> > in the a separate "failing-tests-suite".
> > They should _succeed_, with a comment pointing to the JIRA they
> > reproduce and document.
> > When such an issue gets fixed, the test fails, will be detected and
> > the assertion in question can be inverted. Voila.
>
> And when you have an Exception you add a try/catch? Who guarantees that
> nobody forgets what is right?

By adding a
    fail("this fails as soon as JIRA-9999 is fixed");
following the statement raising the exception in the try block.
As soon as the fix is implemented this "fail" is detected and removed
and the catch block is changed to contain
    fail("no longer raising exception since JIRA-9999 is fixed);

> I never heard of this as a good practice for unit tests. IMO *this*
> would obscure the tests.
> The assertion logic could be more complicated than a single boolean
> assert. (which could be an anti-pattern of course)

But no so complicated to justify polluting the whole test suite with failures.

> The developer working on the fix runs the tests again and again, maybe
> using a debugger. I think this should be done on the original, not on a
> inverted one.

Agreed. But you are talking about work in progress on a working copy.

> Unit tests can be part of the definition. They should not be changed in
> any way as long they are valid.

Not agreed. This is a too dogmatic point of view.

> >From the junit FAQ (okay some kind of TDD related): "When all the tests
> pass, you know you're done!"

+1

> Before changing/inverting anything I would comment out them completely.

Many people do, I think it has downsides.

> > > I propose to accept failing tests in the codebase. Nightly build should
> > > not fail just because tests are failing. Instead we could provide a
> > > testreport in the same folder.
> > > Of course tests ran from ant should not be interrupted by the first
> > > failure. (haltonfailure="no")
> > +1
>
> Maybe we could start a vote on that after discussing.

If there is nobody objecting it would be sufficient to go without a formal vote.

> > > Well, of course this brings some administrative overhead. Maybe we need
> > > a solution to sort our tests using TestSuites.
> > > We could separate the tests known to fail to be warned at once when a
> > > change brings additional tests to fail.
> > -1
>
> Why? Doesn't this avoid the obscuring issue? The regular tests pass and
> you have a binary result. Wouldn't it be nice to have all known-bugs in
> one TestSuite?

>From my experience, TestSuites get outdated very fast because
maintaining them is administrative overhead. I do not use them
anymore.

> > > I don't consider slow tests as bad tests. They are just inappropriate
> > > for a ad hoc results. Even if they took an hour it's okay, but we need
> > > to find a way not to force people to run them all the time.
> >
> > Not agreed. According to my experience unit tests should be fast and
> > easy. Otherwise your unit test setup is flawed or could be simplified.
>
> I agree. That's why I tried to avoid the term "unit test". :-)
>
> > Other tests, of course, for example integration or compliance test
> > could take much more time.
>
> Right. As I started to test which IMAP commands work in the current
> implementation I wrote integration tests. These have proven to be very
> useful.
> If the problem arises, that they are to slow for "every-day-use" we
> should separate, but not change them.

integration tests don't belong into a unit test suite.
per definition, in java, unit tests test (more or less) a single class
("the unit").

  Bernd

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message