[Swan-dev] testing yesterday's tree

[Andrew Cagney]

On 14 May 2018 at 23:44, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> | From: Andrew Cagney <andrew.cagney at gmail.com>
>
> Thanks for looking at this and explaining it.
>
> | However, 6 unresolved is a worry.
>
> What does "unresolved" mean?

See end, but a simple summary is: a human is required

> |  II suspect your domains are dying,
> | do: make kvm-clean
>
> That's what I started off with, immediately before running the tests.

The reason I suspect the domains are dying is the message indicating
output is missing for certain domains - this happens when a domain
didn't manage to boot given a very reasonable amount of time.

The next biggest rock after kvm-clean is kvm-purge and that is what
testing.libreswan.org uses.

--

Dejagnu describes UNRESOLVED (from POSIX 1003.3) as follows:

A test produced indeterminate results. Usually, this means the test
executed in an unexpected fashion. This outcome requires a human to go
over results to determine if the test should have passed or failed.
This message is also used for any test that requires human
intervention because it is beyond the abilities of the testing
framework. Any unresolved test should resolved to PASS or FAIL before
a test run can be considered finished.

Note that for POSIX, each assertion must produce a test result code.
If the test isn’t actually run, it must produce UNRESOLVED rather than
just leaving that test out of the output. This means that you have to
be careful when writing tests to not carelessly use Tcl commands like
return—if you alter the flow of control of the Tcl code you must
insure that every test still produces some result code.

Here are some of the ways a test may wind up UNRESOLVED:

Execution of a test is interrupted.
A test does not produce a clear result. This is usually because there
was an ERROR from DejaGnu while processing the test, or because there
were three or more WARNING messages. Any WARNING or ERROR messages can
invalidate the output of the test. This usually requires a human to
examine the output to determine what really happened – and to improve
the test case.
A test depends on a previous test, which has failed.
The test was set up incorrectly.