[Swan-dev] changes in test results

Thu Feb 21 14:41:23 UTC 2019

PLEASE FIX THESE APPARENTLY TRIVIAL PROBLEMS.

I ran the tests early this morning.  I compared the results with the ones 
from my run a week ago.  Some improvements, some new failures.

First comes my summary analysis of the new problems.  (Heaven knows we
need the same analysis of the old problems.)  Almost all of these
should be fixed right away since they are trivial and generate chaff.

After that, I include the diff of the testing/utils/kvmresults.py
outputs, interspersed with snippits of log file diffs.

(1)

testing/pluto/ikev2-x509-23-no-ca failed east:output-different
testing/pluto/nss-cert-nosecret failed east:output-different west:output-different
testing/pluto/nss-cert-ocsp-02 failed east:output-different
testing/pluto/nss-cert-ocsp-02-ikev2 failed east:output-different
testing/pluto/nss-cert-ocsp-07-nourl failed east:output-different
testing/pluto/nss-cert-ocsp-08-post failed east:output-different
testing/pluto/nss-cert-badca-01 failed east:output-different

Somehow some cert errors are duplicate.

(2) [endemic: not likely to be easy to fix]

testing/pluto/xauth-pluto-14 failed road:output-different
testing/pluto/xauth-pluto-26 failed road:output-different

As always, some data (ping?)  packets get dropped, apparently non-deterministically.

(3)

testing/pluto/newoe-21-liveness-clear failed east:output-different

Timing changed withing log message.  Is this just non-determinism?
Does this need sanitization?

(4)

testing/pluto/newoe-27-replace-sa-auth-authnull failed road:output-different

EVENT_SA_REKEY is now used where EVENT_SA_REPLACE was previously used.
Reference log should be updated.

(6)

testing/pluto/interop-ikev1-strongswan-01-xauth failed east:output-different west:output-different
+strongswan 5.7.1 must be installed

What's this about?  I seem to have 5.7.2 installed.

- should that not be good enough?

- should the message not report what version is actually installed?

	Test requires strongswan 5.7.1 but 5.7.2 is installed

(7)

testing/pluto/nss-cert-nosecret failed east:output-different west:output-different

Looks like a sanitizer change without an update to reference logs.

(8)

testing/pluto/nss-cert-ocsp-03-strict failed west:output-different

Mystery change, probably involving message Ids.

================================================================
< testing/pluto/enumcheck-01 failed west:output-different
> testing/pluto/enumcheck-01 passed

< testing/pluto/ikev2-algo-06-aes-aes_xcbc failed east:output-different west:output-different
> testing/pluto/ikev2-algo-06-aes-aes_xcbc passed

< testing/pluto/ikev2-algo-ike-sha2-01 failed east:output-different west:output-different
< testing/pluto/ikev2-algo-ike-sha2-02 failed east:output-different west:output-different
< testing/pluto/ikev2-algo-ike-sha2-03 failed east:output-different west:output-different
> testing/pluto/ikev2-algo-ike-sha2-01 passed
> testing/pluto/ikev2-algo-ike-sha2-02 passed
> testing/pluto/ikev2-algo-ike-sha2-03 passed

< testing/pluto/ikev2-x509-23-no-ca passed
> testing/pluto/ikev2-x509-23-no-ca failed east:output-different
testing/pluto/ikev2-x509-23-no-ca/OUTPUT/east.console.diff
 "ikev2-westnet-eastnet-x509-cr" #1: ERROR: Peer's Certificate issuer is not recognized.
+"ikev2-westnet-eastnet-x509-cr" #1: ERROR: Peer's Certificate issuer is not recognized.

< testing/pluto/newoe-21-liveness-clear passed
> testing/pluto/newoe-21-liveness-clear failed east:output-different
testing/pluto/newoe-21-liveness-clear/OUTPUT/east.console.diff
-"private-or-clear#192.1.3.0/24"[1] ...192.1.3.209 #2: liveness_check - peer 192.1.3.209 has not responded in 12 seconds, with a timeout of 10, taking action:clear
+"private-or-clear#192.1.3.0/24"[1] ...192.1.3.209 #2: liveness_check - peer 192.1.3.209 has not responded in 11 seconds, with a timeout of 10, taking action:clear

< testing/pluto/newoe-27-replace-sa-auth-authnull passed
> testing/pluto/newoe-27-replace-sa-auth-authnull failed road:output-different
testing/pluto/newoe-27-replace-sa-auth-authnull/OUTPUT/road.console.diff
-000 #1: "private-or-clear#192.1.2.0/24"[1] ...192.1.2.23:500 STATE_PARENT_I2 (sent v2I2, expected v2R2); EVENT_SA_REKEY in XXs; idle;
+000 #1: "private-or-clear#192.1.2.0/24"[1] ...192.1.2.23:500 STATE_PARENT_I2 (sent v2I2, expected v2R2); EVENT_SA_REPLACE in XXs; idle;

< testing/pluto/newoe-18-poc-cop-port22-transport failed road:output-different
> testing/pluto/newoe-18-poc-cop-port22-transport passed

< testing/pluto/ikev2-liveness-09 failed west:output-different
> testing/pluto/ikev2-liveness-09 passed

< testing/pluto/ikev2-child-03-dh-none-pfs-no failed west:output-different
< testing/pluto/ikev2-child-04-dh-none-pfs-yes failed west:output-different
> testing/pluto/ikev2-child-03-dh-none-pfs-no passed
> testing/pluto/ikev2-child-04-dh-none-pfs-yes passed

< testing/pluto/xauth-pluto-14 passed
> testing/pluto/xauth-pluto-14 failed road:output-different
testing/pluto/xauth-pluto-14/OUTPUT/road.console.diff
-006 #2: "modecfg-road-east", username=use3, type=ESP, add_time=1234567890, inBytes=336, outBytes=336
+006 #2: "modecfg-road-east", username=use3, type=ESP, add_time=1234567890, inBytes=716, outBytes=604

< testing/pluto/xauth-pluto-16 failed road:output-different
> testing/pluto/xauth-pluto-16 passed

< testing/pluto/xauth-pluto-26 passed
> testing/pluto/xauth-pluto-26 failed road:output-different
testing/pluto/xauth-pluto-26/OUTPUT/road.console.diff
-4 packets transmitted, 4 received, 0% packet loss, time XXXX
+4 packets transmitted, 3 received, 25% packet loss, time XXXX

> testing/pluto/interop-ikev1-strongswan-01-xauth failed east:output-different west:output-different
testing/pluto/interop-ikev1-strongswan-01-xauth/OUTPUT/east.console.diff
+strongswan 5.7.1 must be installed

< testing/pluto/l2tp-02 failed north:output-different
> testing/pluto/l2tp-02 passed

< testing/pluto/nss-cert-nosecret passed
> testing/pluto/nss-cert-nosecret failed east:output-different west:output-different
testing/pluto/nss-cert-nosecret/OUTPUT/east.console.diff
-000 End certificate "east" - SN: 0xXXXX
+000 End certificate "east" - SN: 0xXX
and many more

< testing/pluto/nss-cert-ocsp-02 passed
> testing/pluto/nss-cert-ocsp-02 failed east:output-different
testing/pluto/nss-cert-ocsp-02/OUTPUT/east.console.diff
 "nss-cert-ocsp" #1: certificate revoked!
+"nss-cert-ocsp" #1: certificate revoked!

< testing/pluto/nss-cert-ocsp-02-ikev2 passed
> testing/pluto/nss-cert-ocsp-02-ikev2 failed east:output-different
testing/pluto/nss-cert-ocsp-02-ikev2/OUTPUT/east.console.diff
 "nss-cert-ocsp" #1: certificate revoked!
+"nss-cert-ocsp" #1: certificate revoked!

< testing/pluto/nss-cert-ocsp-03-strict passed
> testing/pluto/nss-cert-ocsp-03-strict failed west:output-different
testing/pluto/nss-cert-ocsp-03-strict/OUTPUT/west.console.diff
 1v1 "nss-cert-ocsp" #1: STATE_MAIN_I3: sent MI3, expecting MR3
-003 "nss-cert-ocsp" #1: ignoring informational payload INVALID_ID_INFORMATION, msgid=00000000, length=12
-003 "nss-cert-ocsp" #1: received and ignored notification payload: INVALID_ID_INFORMATION

< testing/pluto/nss-cert-ocsp-07-nourl passed
> testing/pluto/nss-cert-ocsp-07-nourl failed east:output-different
testing/pluto/nss-cert-ocsp-07-nourl/OUTPUT/east.console.diff
 "nss-cert-ocsp" #1: certificate revoked!
+"nss-cert-ocsp" #1: certificate revoked!

< testing/pluto/nss-cert-ocsp-08-post passed
> testing/pluto/nss-cert-ocsp-08-post failed east:output-different
testing/pluto/nss-cert-ocsp-08-post/OUTPUT/east.console.diff
  grep -i "certificate revoked" /tmp/pluto.log
 "nss-cert-ocsp" #1: certificate revoked!
+"nss-cert-ocsp" #1: certificate revoked!
 east #
  grep ERROR /tmp/pluto.log
+"nss-cert-ocsp" #1: ERROR: Peer's Certificate has been revoked.
 "nss-cert-ocsp" #1: ERROR: Peer's Certificate has been revoked.

< testing/pluto/nss-cert-badca-01 passed
> testing/pluto/nss-cert-badca-01 failed east:output-different
testing/pluto/nss-cert-badca-01/OUTPUT/east.console.diff
 "nss-cert" #1: ERROR: Issuer certificate is invalid.
+"nss-cert" #1: ERROR: Issuer certificate is invalid.