<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 5 Sept 2021 at 09:44, D. Hugh Redelmeier <<a href="mailto:hugh@mimosa.com">hugh@mimosa.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I ran the tests suite last night on two machines. It used to be that one <br>
would expect different results from two runs "due to non-determanism", <br>
even on the same machine.<br>
<br>
I got the same results on two machines.  That's either good news or luck.  <br>
This is probably the result of hard work: thanks!<br></blockquote><div><br></div><div>It's a bit of both, for instance, these two runs had 36 failures:</div><div>  <a href="https://testing.libreswan.org/?run=v4.5-36-gcb013f4154-main&run=v4.5-35-g6a4754e1fd-main">https://testing.libreswan.org/?run=v4.5-36-gcb013f4154-main&run=v4.5-35-g6a4754e1fd-main</a></div><div>But yes, things are definitely better.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
kvmrunner 6:22:43.06:   stats/skiptest/ignored/status!=good: 9<br>
kvmrunner 6:22:43.06:   stats/wip/ignored/status!=good: 79<br>
kvmrunner 6:22:43.06:   tests/good/errors/EXPECTATION: 3<br>
kvmrunner 6:22:43.06:   tests/good/errors/output-different: 43<br>
kvmrunner 6:22:43.06:   tests/good/results/failed: 46<br>
kvmrunner 6:22:43.06:   tests/good/results/passed: 832<br>
kvmrunner 6:22:43.06:   tests/skiptest/results/untested: 9<br>
kvmrunner 6:22:43.06:   tests/wip/results/untested: 79<br>
kvmrunner 6:22:43.06:   total: 966<br>
kvmrunner 6:22:43.06:   total/failed: 46<br>
kvmrunner 6:22:43.06:   total/ignored: 88<br>
kvmrunner 6:22:43.06:   total/passed: 832<br>
<br>
The machines are essentially identical except that one has an 8G of RAM <br>
and an SSD and the other has 16G of RAM and a HDD.  The different disk <br>
speeds might tend to amplify non-determinism.  The run times were 5:34 and <br>
6:23.<br>
_______________________________________________<br>
Swan-dev mailing list<br>
<a href="mailto:Swan-dev@lists.libreswan.org" target="_blank">Swan-dev@lists.libreswan.org</a><br>
<a href="https://lists.libreswan.org/mailman/listinfo/swan-dev" rel="noreferrer" target="_blank">https://lists.libreswan.org/mailman/listinfo/swan-dev</a><br>
</blockquote></div></div>