[Swan-dev] consider unchecked output UNRESOLVED

[Andrew Cagney]

I've finally pushed this change.

On 19 August 2016 at 15:13, Andrew Cagney <andrew.cagney at gmail.com> wrote:
> Currently, when checking the results from a test, missing reference
> output is ignored.  For instance, the test:
>
> $ ls -1 testing/pluto/interop-ikev2-racoon-01-noconn/{,OUTPUT/}
> testing/pluto/interop-ikev2-racoon-01-noconn/:
> description.txt
> eastinit.sh
> east-racoon
> final.sh
> OUTPUT
> testparams.sh
> west.conf
> westinit.sh
> westrun.sh
>
> testing/pluto/interop-ikev2-racoon-01-noconn/OUTPUT/:
> debug.log
> east.console.txt
> east.console.verbose.txt
> pluto.east.log
> RESULT
> west.console.txt
> west.console.verbose.txt
>
> has no reference output at all, yet it is considered a "good" test and
> passes.  Perhaps the intent was to detect a core dump, but that isn't
> robust.
>
> I believe the rationale for this behaviour was that, in some cases,
> getting output to match is "just too hard" (I can kind of believe it);
> however I think a better way to deal with it would be to explicitly
> sanitize away the output that should be ignored.  For instance, by
> putting:
>
> : =================== end ==================
>
> in the relevant *init.sh file.
>
> Ignoring "nic", there are ~18 "good" tests that have this problem; I
> think they be changed to "wip".
> In addition there are ~165 "good" tests that use "nic" (nic's output
> is never checked!)', I think we can brute force them to just ignore
> "nic"s output
>
> Andrew
>
> PS: Why?  I've tests that need to check "nic"s output, and maybe doing
> it isn't exactly robust :-/