[Swan-dev] two test runs, on different machines, matched

[Andrew Cagney]

On Sun, 5 Sept 2021 at 09:44, D. Hugh Redelmeier <hugh at mimosa.com> wrote:

> I ran the tests suite last night on two machines. It used to be that one
> would expect different results from two runs "due to non-determanism",
> even on the same machine.
>
> I got the same results on two machines.  That's either good news or luck.
> This is probably the result of hard work: thanks!
>

It's a bit of both, for instance, these two runs had 36 failures:

https://testing.libreswan.org/?run=v4.5-36-gcb013f4154-main&run=v4.5-35-g6a4754e1fd-main
But yes, things are definitely better.


>
> kvmrunner 6:22:43.06:   stats/skiptest/ignored/status!=good: 9
> kvmrunner 6:22:43.06:   stats/wip/ignored/status!=good: 79
> kvmrunner 6:22:43.06:   tests/good/errors/EXPECTATION: 3
> kvmrunner 6:22:43.06:   tests/good/errors/output-different: 43
> kvmrunner 6:22:43.06:   tests/good/results/failed: 46
> kvmrunner 6:22:43.06:   tests/good/results/passed: 832
> kvmrunner 6:22:43.06:   tests/skiptest/results/untested: 9
> kvmrunner 6:22:43.06:   tests/wip/results/untested: 79
> kvmrunner 6:22:43.06:   total: 966
> kvmrunner 6:22:43.06:   total/failed: 46
> kvmrunner 6:22:43.06:   total/ignored: 88
> kvmrunner 6:22:43.06:   total/passed: 832
>
> The machines are essentially identical except that one has an 8G of RAM
> and an SSD and the other has 16G of RAM and a HDD.  The different disk
> speeds might tend to amplify non-determinism.  The run times were 5:34 and
> 6:23.
> _______________________________________________
> Swan-dev mailing list
> Swan-dev at lists.libreswan.org
> https://lists.libreswan.org/mailman/listinfo/swan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20210905/3442aac4/attachment.html>