[Swan-dev] consider unchecked output UNRESOLVED

[Andrew Cagney]

Currently, when checking the results from a test, missing reference
output is ignored.  For instance, the test:

$ ls -1 testing/pluto/interop-ikev2-racoon-01-noconn/{,OUTPUT/}
testing/pluto/interop-ikev2-racoon-01-noconn/:
description.txt
eastinit.sh
east-racoon
final.sh
OUTPUT
testparams.sh
west.conf
westinit.sh
westrun.sh

testing/pluto/interop-ikev2-racoon-01-noconn/OUTPUT/:
debug.log
east.console.txt
east.console.verbose.txt
pluto.east.log
RESULT
west.console.txt
west.console.verbose.txt

has no reference output at all, yet it is considered a "good" test and
passes.  Perhaps the intent was to detect a core dump, but that isn't
robust.

I believe the rationale for this behaviour was that, in some cases,
getting output to match is "just too hard" (I can kind of believe it);
however I think a better way to deal with it would be to explicitly
sanitize away the output that should be ignored.  For instance, by
putting:

: =================== end ==================

in the relevant *init.sh file.

Ignoring "nic", there are ~18 "good" tests that have this problem; I
think they be changed to "wip".
In addition there are ~165 "good" tests that use "nic" (nic's output
is never checked!)', I think we can brute force them to just ignore
"nic"s output

Andrew

PS: Why?  I've tests that need to check "nic"s output, and maybe doing
it isn't exactly robust :-/