[Swan-dev] changes in test results

[Paul Wouters]

On Thu, 21 Feb 2019, D. Hugh Redelmeier wrote:

> PLEASE FIX THESE APPARENTLY TRIVIAL PROBLEMS.

SADLY NOT ALL OF THEM ARE TRIVIAL :)

Let me comment on the ones I can from the top of my head

> testing/pluto/nss-cert-nosecret failed east:output-different west:output-different

This is a bug. The test case shows the problem. A code fix is required,
and not trivial. This is a reminder we need to fix this (let's not start
another 'we need anothe bug category' discussion now.

> testing/pluto/nss-cert-ocsp-02 failed east:output-different
> testing/pluto/nss-cert-ocsp-02-ikev2 failed east:output-different
> testing/pluto/nss-cert-ocsp-07-nourl failed east:output-different
> testing/pluto/nss-cert-ocsp-08-post failed east:output-different

Usually due to not regenerating certificates?

> Somehow some cert errors are duplicate.

That should probably just be cleaned up in code.

> testing/pluto/newoe-21-liveness-clear failed east:output-different
>
> Timing changed withing log message.  Is this just non-determinism?
> Does this need sanitization?

all dpd/liveness tests need a better approach to reduce false positives.
They are all too fragile.

> (4)
>
> testing/pluto/newoe-27-replace-sa-auth-authnull failed road:output-different
>
> EVENT_SA_REKEY is now used where EVENT_SA_REPLACE was previously used.
> Reference log should be updated.

If that's the only thing sure. This test did indicate a recent
regression that was fixed, so care should be taken it is still properly
working.

> testing/pluto/interop-ikev1-strongswan-01-xauth failed east:output-different west:output-different
> +strongswan 5.7.1 must be installed
>
> What's this about?  I seem to have 5.7.2 installed.

A few days ago, fedora updates pushed 5.7.2. I will bump the versions
required.

> - should that not be good enough?

It is not about good enough. When people use different versions, they
keep overwriting each others test results with minor variances. That
is why I forced a single version or completely fail the test.

> - should the message not report what version is actually installed?
>
> 	Test requires strongswan 5.7.1 but 5.7.2 is installed

patches accepted :)

Paul