[Swan-dev] drop ipsec-auto-up.n.sed

[Andrew Cagney]

On Mon, 28 Sep 2020 at 21:53, Paul Wouters <paul at nohats.ca> wrote:

> On Mon, 28 Sep 2020, Andrew Cagney wrote:
>
> > I'm planning on removing the sanitizer ipsec-auto-up.n.sed.
>
> I understand why, but I also understand's Antony's point.
>
> To make it harder, once I rewrite the tests to also be documentation,
> we have an additionally issue of adding weird things to the configs
> people might copy. Like slow retransmit timeouts or impairs.
>

slow-retransmits should be deleted (I thought I had, I see I hadn't)
Your suspicion that IKEv1 didn't suppress retransmits turned out to be
because the tests in question weren't trying to supress retransmits.


>
> I'm also a bit nervous if 90% of our tests use --impair
> delete-on-retransmit At one point are we still testing "regular operation"
> code with our tests.
>

Two types of tests:
1. regression tests - these should be deterministic - someone reports a bug
or we add a feature and we add a test for it
2. stress testing; the whole point of these is to be non-deterministic
but we're discussing deterministic tests here


> Antony's filter in ipsec-auto-up.n.sed does address most of these
> issues.


>
This assertion is not true.  Please look at the examples I posted.  It
silently removes important contextual information from test output - namely
that a retransmit occurred - while leaving behind the consequences.  The
upshot is that a retransmit still causes the test to fail; only that the
retransmit happened is obscured.

With the sanitizer disabled I got roughly the same number of fails.




> But other issues still remain, such as retransmits causing
> different traffic counters in the output and causing false positives.


> So, I guess I have no good answers or strong opinions here. Just
> concerns :)
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20200928/fd36e518/attachment.html>