[Swan-dev] testing/pluto/ikev2-03-basic-rawrsa-ckaid

[Paul Wouters]

On Sat, 2 Feb 2019, D. Hugh Redelmeier wrote:

> | It’s a bug but an old one. It might be triggered now with some secrets file install change
>
> So: is this test known to fail?

For this one I am not entirely sure. For some others we do know that
they are failing because work needs to be done.

> If it is, it is a waste of time and a cognative burden for each person
> testing.

We have gone through a few rounds of discussions on this. There are no
good options.

> - we need an announced category of "expected failures"

It won't help, because people will always ignore it and never fix it.

> - we need reference logs reflecting such bugs so that when something
>  changes in the output we get a signal.

If we ship with known bugs, we can have the test system show them as
bugs?

> - if the expected failure creates non-deterministic results, we need
>  to declare that too

non-deterministic is something no one wants. It just happens due to a
number of known and unknown reasons.

> The alternative is to ignore any test that failed "before".  That's
> negligent, but it's what most of us do.

Yes, there are no good options, other than getting more resources to fix
things :P

Paul