[Swan-dev] two test runs, on different machines, matched

Andrew Cagney andrew.cagney at gmail.com
Sun Sep 26 20:10:45 UTC 2021


  total: 967
  total/failed: 29
  total/ignored: 91
  total/passed: 847
ikev2-32-nat-rw-rekey failed east:output-different road:output-different
ikev2-48-nat-cp-start failed road:output-different
newoe-20-ipv6 failed east:output-different road:output-different
newoe-27-replace-sa-auth-authnull failed road:output-different
certoe-12-nat-server failed road:output-different
certoe-14-poc-del-slash32 failed east:EXPECTATION
certoe-15-poc-east-west failed east:EXPECTATION
certoe-17-asymmetric-cert-nat-testing-client-crash failed
east:output-different road:output-different
certoe-18-pass-then-go-slash32-keyingtries1 failed road:output-different
newoe-18-poc-cop-port22-both-reorder failed east:output-different
road:output-different
dpd-01 failed west:output-different
dpd-02 failed east:output-different
ikev2-liveness-05 failed west:output-different
ikev2-liveness-06 failed east:output-different road:output-different
ikev2-liveness-07 failed road:output-different
ikev2-child-rekey-06-deadlock failed west:output-different
ikev2-child-rekey-09-windows failed road:output-different
delete-sa-05 failed west:output-different
ikev1-rw-multiple-subnets failed east:output-different
ikev1-rw-multiple-subnets-split failed east:output-different
road:output-different
ikev2-rw-multiple-subnets failed east:output-different
ikev2-rw-multiple-subnets-2 failed east:output-different
ikev2-rw-multiple-subnets-3 failed east:output-different
ikev2-rw-multiple-subnets-4-mismatch failed east:output-different
road:output-different
ikev2-labeled-ipsec-04-no-label-on-initiator failed east:output-different
nss-cert-crl-03-strict failed east:output-different
ikev2-mobike-03 failed road:EXPECTATION
ikev2-systemrole-03-certs failed west:output-different
ikev2-crossing-streams-03-iface failed east:output-different
road:output-different

On Thu, 9 Sept 2021 at 20:41, Andrew Cagney <andrew.cagney at gmail.com> wrote:

>
>
> On Sun, 5 Sept 2021 at 11:17, Paul Wouters <paul at nohats.ca> wrote:
>
>> On Sun, 5 Sep 2021, Andrew Cagney wrote:
>>
>> > On Sun, 5 Sept 2021 at 09:44, D. Hugh Redelmeier <hugh at mimosa.com>
>> wrote:
>> >       I ran the tests suite last night on two machines. It used to be
>> that one
>> >       would expect different results from two runs "due to
>> non-determanism",
>> >       even on the same machine.
>> >
>> >       I got the same results on two machines.  That's either good news
>> or luck.
>> >       This is probably the result of hard work: thanks!
>> >
>> >
>> > It's a bit of both, for instance, these two runs had 36 failures:
>> >
>> https://testing.libreswan.org/?run=v4.5-36-gcb013f4154-main&run=v4.5-35-g6a4754e1fd-main
>> > But yes, things are definitely better.
>>
>> I think Andrew contributed most of this part in the last two months
>> actually. But good to
>> see indeed! Next up, fixing liveness/dpd and WIP tests :)
>>
>
>  Time to move the tests that won't pass to wip?
>
>
>> Paul
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20210926/6a704bb0/attachment.html>


More information about the Swan-dev mailing list