[Swan-dev] good tests that never pass

[Andrew Cagney]

I'm getting ready to push a functional change to the IKEv2 responder: when
auth succeeds but the first child fails the IKE SA hangs around.  I expect
this will perturb the test results in some pretty weird ways so before
doing this I'd like to be working with a very clean test result (i.e. 0
failures).

Which brings me to my problem. A look at https://testing.libreswan.org/
shows things are stuck at 40 fails.  A closer look shows:

- 30 or so of those just refuse to pass (as in after 10 attempts they still
fail)
- the remainder are racy but do pass if given enough encouragement; for
instance sometimes fail due to a LEAK race during shutdown

I'd like to change this.  In order of preference:

- change the tests that simply refuse to pass to WIP so that they aren't
part of a default test run
- move the tests that refuse to pass to the end of TESTLIST so they are run
after the tests that should pass

I don't see a good reason to remove the tests that flip flop.  If anything,
with the broken tests out the way it will be easier to give those some tlc.

Of course this suggestion isn't new.  The counter argument is that "we"
should instead be fixing these tests.  And the counter counter argument is
that if these tests were important someone would have fixed them already.

(regardless I'm going to start using a custom TESTLIST that excludes tests
I consider broken)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20210513/a3a4cf08/attachment.html>