[Swan-dev] a scan of failing tests

[D. Hugh Redelmeier]

I scanned the output of kvmresults.py and added an annotation for each
failing test.  Here's the result.

I hope that this helps others home in on tests that they can rehabilitate.

First, only the failures.  You'll need a wide terminal to view this.

basic-pluto-00 failed west:output-different <== no hope of working
ikev2-delete-02 failed east:output-different west:output-different <== reference logs obsolete
ikev2-12-x509-ikev1-rw failed west:output-different <=== leak reported
ikev2-12-transport-psk failed east:output-different west:output-different <== local policy requires Transport Mode but peer requires required Tunnel Mode
ikev2-32-nat-rw-rekey failed east:EXPECTATION road:output-different <== lease newly reported???
newoe-25-cat-2 failed road:output-different <== west conns not loaded?
newoe-25-cat-3-4-way failed east:output-different north:output-different road:output-different west:output-different <=== road diagram changed
newoe-25-cat-4 failed east:output-different road:output-different west:output-different <== not clear
newoe-25-cat-5 failed east:output-different road:output-different <=== not clear
dpd-01 failed west:output-different <== timing of SA?
dpd-04 failed east:output-different west:output-different <== address and port changes?
dpd-05 failed west:output-different <== port change
dpd-06 failed west:output-different <== new bad iptables rules?
ikev2-liveness-05 failed east:output-different west:output-different <== no longer KLIPS?
ikev2-liveness-06 failed east:EXPECTATION,output-different road:output-different <== no longer KLIPS?
ikev2-liveness-07 failed east:EXPECTATION,output-different road:output-different <== no longer KLIPS?
ikev2-liveness-08-drop failed east:output-different west:output-different <== no longer KLIPS?
delete-sa-03 failed east:output-different west:output-different <== refhim
x509-pluto-02 failed north:output-different <== no outBytes
x509-pluto-frag-01 failed road:output-different <= retransmission message
nat-pluto-04 failed east:output-different road:output-different <== ?
ikev1-algo-ike-aes-02 failed east:output-different west:output-different <== no longer KLIPS
psk-pluto-02 incomplete east:output-different road:output-different <== INVALID_HASH_INFORMATION ?
xauth-pluto-12 failed east:output-different road:output-different <== no longer KLIPS
xauth-pluto-15 failed east:output-different road:output-different <== no longer KLIPS
xauth-pluto-17 failed east:output-different <== swapped IP in XFRM ?
xauth-pluto-18 failed east:output-different <== swapped IP in XFRM ?
basic-pluto-12 failed east:output-different west:output-different <== lots of additional activity
interop-ikev2-strongswan-16-ah-initiator-sha512 failed west:output-different <== cert req changed?
interop-ikev2-strongswan-17-ah-initiator-sha256 failed west:output-different <== cert req changed?
interop-ikev1-strongswan-15-twofish failed east:output-different west:output-different <== message ignored because it contains an unexpected payload type (ISAKMP_NEXT_HASH)
interop-ikev1-strongswan-16-serpent failed east:output-different west:output-different <== message ignored because it contains an unexpected payload type (ISAKMP_NEXT_HASH)
ipv6-tunnel-mode-02-netkey-netkey failed east:output-different west:output-different <== failure to communicate
ipv6-tunnel-mode-04-rw failed east:output-different road:output-different <== failure to communicate
interop-ikev2-racoon-02-psk-responder failed east:output-different west:output-different <== ?
interop-ikev2-strongswan-03-psk-initiator failed west:output-different <== cert request different [but PSK??]
interop-ikev2-strongswan-04-x509-responder failed west:output-different <== failed to negoriate
interop-ikev2-strongswan-07-strongswan failed west:output-different <== cert request different
interop-ikev2-strongswan-13-ah-initiator failed west:output-different <== cert request different
interop-ikev2-strongswan-14-delete-sa failed west:output-different <== cert request different
interop-ikev2-strongswan-15-create_child_sa incomplete east:output-different west:output-different <== ss cannot parse ls message
interop-ikev2-strongswan-17-delete-sa-responder failed west:output-different <== cert request different
interop-ikev2-strongswan-27-fragmentation failed west:output-different <== negotiation failed
interop-ikev2-strongswan-28-reauth failed west:output-different <== cert request different
dnssec-pluto-01 failed west:output-different <== westnet-eastnet-etc-hosts-auto-add missing
l2tp-01 failed east:output-different north:output-different <== test changed?  No longer KLIPS?
l2tp-02 failed east:output-different north:output-different <== test changed?  No longer KLIPS?
fips-08-ikev2-x509 failed east:output-different west:output-different <== test changed?  No longer KLIPS?
nss-cert-04 failed west:output-different <== ignoring retry?
nss-cert-05 failed east:EXPECTATION west:output-different <== ignoring retry?
nss-cert-08-mismatch failed west:EXPECTATION,output-different <== logging seems to have changed
nss-cert-chain-02 failed west:output-different <== retransmission different
nss-cert-crl-02 failed west:output-different <== duplicate packet
nss-cert-nosecret failed east:output-different west:output-different <== certs or keys different?
nss-cert-ocsp-01-strict failed west:output-different <== negotiation failed
nss-cert-ocsp-02 failed east:output-different west:output-different <== no "certificate revoked"; cert accepted!!
nss-cert-ocsp-02-ikev2 failed east:output-different west:output-different <== no "certificate revoked"; cert accepted!!
nss-cert-ocsp-02-strict failed east:output-different west:output-different <== no "certificate revoked"; cert rejected for a different reason
nss-cert-ocsp-03-strict failed west:output-different <== lots more retransmissions
nss-cert-ocsp-04 failed west:output-different <== negotiation succeeded (mistake??)
nss-cert-ocsp-04-strict failed west:output-different <= old: INVALID_ID_INFORMATION, new: INVALID_KEY_INFORMATION ???
nss-cert-ocsp-05-strict failed west:output-different <== negotiation failure when success is expected
nss-cert-ocsp-06 failed west:output-different <== negotiation success when failure expected
nss-cert-ocsp-07-nourl failed east:output-different west:output-different <== no "certificate revoked!", negotiation unexpectedly succeeded




The rest is the whole original output of kvmresults.py, with annotations.

kvmresults 0.057: ****** invalid test klips-ikev2-algo-sha2-0: directory not found: testing/pluto/klips-ikev2-algo-sha2-0
basic-pluto-00 failed west:output-different <== no hope of working
basic-pluto-01 passed
basic-pluto-02 passed
basic-pluto-03 passed
basic-pluto-04 passed east:output-unchecked west:output-unchecked
basic-pluto-07-sourceip passed east:output-unchecked west:output-unchecked
netkey-audit-01 passed
ikev1-01-fuzzer passed
ikev2-01-fallback-ikev1 passed
ikev2-02-responder-send-notify passed
ikev2-03-basic-rawrsa passed
ikev2-04-basic-x509 passed
ikev2-04-basic-x509-ckaid passed
ikev2-05-basic-psk passed
ikev2-dcookie-01 passed
ikev2-dcookie-02 passed east:output-unchecked west:output-unchecked
ikev2-dcookie-03 passed east:output-unchecked west:output-unchecked
ikev2-08-delete-notify passed
ikev2-delete-01 passed
ikev2-delete-02 failed east:output-different west:output-different <== reference logs obsolete
ikev2-09-rw-rsa passed
ikev2-11-simple-psk passed
ikev2-12-x509-ikev1 passed
ikev2-12-x509-ikev1-rw failed west:output-different <=== leak reported
ikev2-12-transport-psk failed east:output-different west:output-different <== local policy requires Transport Mode but peer requires required Tunnel Mode
ikev2-13-ah passed
ikev2-14-missing-ke passed
ikev2-15-fuzzer passed
ikev2-17-rekey-ipsec passed
ikev2-18-x509-alias incomplete east:EXPECTATION,output-unchecked road:EXPECTATION,output-unchecked
ikev2-19-x509-auto-start passed east:output-unchecked road:output-unchecked
ikev2-31-nat-rw-no-rekey passed east:EXPECTATION
ikev2-32-nat-rw-rekey failed east:EXPECTATION road:output-different <== lease newly reported???
ikev2-30-rw-no-rekey passed
ikev2-41-rw-replace failed road:output-different
ikev2-42-rw-replace-responder failed road:output-different
ikev2-45-impair-gx-01 passed
ikev2-45-impair-gx-02 failed west:EXPECTATION,output-different
ikev2-46-basic-psk-netkey passed
ikev2-47-priority passed
ikev2-48-nat-cp passed
ikev2-49-hub-spoke failed north:output-unchecked west:output-different
ikev2-50-propnum passed
ikev2-51-rw-nat-ikeport failed east:output-different road:output-different
ikev2-25-rw-nat failed road:output-different
ikev2-29-no-rekey failed east:output-different west:output-different
ikev2-ddns-01 passed
ikev2-ddns-02 failed east:output-different west:output-different
ikev1-cryptoload-01 passed
ikev1-cryptoload-00 passed
ikev2-03-basic-rawrsa-nhelpers0 passed
ikev2-04-basic-x509-nhelpers0 passed
ikev2-05-basic-psk-nhelpers0 passed
ikev2-04-basic-x509-nsspw passed
ikev2-x509-01 passed
ikev2-algo-01-modp2048-initiator passed
ikev2-algo-02-modp2048-responder passed
ikev2-algo-03-aes-ccm passed
ikev2-algo-04-aes-gcm256 passed
ikev2-algo-05-aes-default passed
ikev2-algo-06-aes-aes_xcbc passed
ikev2-algo-07-aes_ctr passed
ikev2-algo-08-cast passed
ikev2-algo-09-camellia passed
ikev2-algo-10-aes-gcm128 passed
ikev2-algo-11-gcm-prop2 passed
netkey-algo-aes_gcm-01 passed
netkey-algo-aes_gcm-02 passed
netkey-algo-aes_gcm-03 passed
netkey-algo-aes_ccm-01 passed
ikev2-algo-sha2-01 passed
ikev2-algo-sha2-02 passed
ikev2-algo-sha2-03 passed
ikev2-algo-sha2-04 passed
ikev2-algo-sha2-05 passed
ikev1-algo-esp-sha2-05 passed
ikev2-algo-sha2-06 failed west:output-different
ikev2-algo-sha2-08 passed
ikev1-algo-esp-sha2-01-netkey-klips passed
ikev1-algo-esp-sha2-02-netkey-klips passed
ikev2-algo-ike-sha2-01 passed
ikev2-algo-ike-sha2-02 passed
ikev2-algo-ike-sha2-03 passed
ikev1-algo-ike-sha2-01 passed
ikev1-algo-ike-sha2-02 passed
ikev1-algo-ike-sha2-03 passed
ikev1-algo-04-mismatch passed
ikev1-algo-05-3des-sha2 passed
ikev2-algo-esn-01 failed west:output-different
ikev2-algo-esn-02 passed east:output-unchecked west:output-unchecked
ikev2-algo-esn-03 failed west:output-different
ikev2-algo-esn-04 failed west:output-different
ikev2-algo-esn-05 failed west:output-different
interop-ikev2-strongswan-30-esn passed
ikev2-major-version-initiator passed
ikev2-major-version-responder passed
ikev2-minor-version-initiator passed
ikev2-minor-version-responder passed
ikev2-isakmp-reserved-flags-01 passed
ikev2-isakmp-reserved-flags-02 passed
ikev2-payload-reserved-flags-01 passed
ikev2-allow-narrow-01 failed east:output-different west:EXPECTATION,output-different
ikev2-allow-narrow-02 failed west:EXPECTATION,output-different
ikev2-allow-narrow-03 failed west:EXPECTATION,output-different
ikev2-allow-narrow-04 passed
ikev2-allow-narrow-05 passed
ikev2-allow-narrow-06 passed
ikev2-allow-narrow-07 passed
ikev2-invalid-ke-01-invalid-modp passed
ikev2-invalid-ke-02-wrong-modp passed
ikev2-invalid-ke-03-default-initiator passed
ikev2-invalid-ke-04-default-responder passed
ikev1-impair-gx-01 passed
ikev1-impair-gx-02 incomplete east:output-unchecked west:output-unchecked
ikev2-no-nhelpers-01 passed
ikev2-auth-null-01 passed
ikev1-switchnat-01 passed east:output-unchecked road:output-unchecked
ikev1-hub-spoke passed north:output-unchecked
pluto-ipcmp-01 passed
ikev2-38-psk-65bytes passed
ikev2-39-psk-64bytes passed
nflog-01-global failed west:output-different
nflog-02-conn failed west:output-different
nflog-03-conns failed east:output-different west:output-different
netkey-vti-01 failed east:output-different west:output-different
netkey-vti-02 passed
netkey-vti-03 passed east:output-unchecked west:output-unchecked
netkey-tfc-01 passed
netkey-tfc-02 passed
netkey-tfc-03 passed
ikev2-43-init-retransmit passed
ikev2-frag-01 passed
ikev2-frag-02-ipv6 failed east:output-different west:output-different
dynamic-iface-01 failed west:output-different
newoe-01-whack passed
newoe-03-oeself passed
newoe-04-pass-pass passed
newoe-05-hold-pass passed
newoe-06-prio passed
newoe-07-ike-replace-initiator passed
newoe-07-ike-replace-initiator-esp failed road:output-different
newoe-08-ike-replace-responder passed
newoe-08-restart passed road:output-whitespace
newoe-09-mutual passed
newoe-10-expire-inactive passed
newoe-10-expire-inactive-ike passed
newoe-11-failike passed
newoe-12-clear passed
newoe-13-block passed
newoe-14-clear-in-block-clear passed
newoe-15-portpass passed
newoe-16-pass-hold passed
newoe-17-block-in-clear-clear passed east:output-unchecked road:output-unchecked
newoe-18-poc-poc passed
newoe-18-poc-cop passed
newoe-18-poc-private passed
newoe-18-poc-clear passed
newoe-18-poc-block failed road:output-different
newoe-18-cop-poc passed
newoe-18-cop-cop passed
newoe-18-cop-private passed
newoe-18-cop-clear passed
newoe-18-cop-block passed
newoe-18-private-poc passed
newoe-18-private-cop passed
newoe-18-private-private passed
newoe-18-private-clear passed
newoe-18-private-block passed
newoe-18-clear-poc passed
newoe-18-clear-cop passed
newoe-18-clear-private passed
newoe-18-clear-clear passed
newoe-18-clear-block passed
newoe-18-block-poc passed
newoe-18-block-cop passed
newoe-18-block-private passed
newoe-18-block-clear passed
newoe-18-block-block passed
newoe-18-poc-blockall passed
newoe-18-private-clearall passed
newoe-18-private-copall passed
newoe-18-private-pocall passed
newoe-19-poc-poc-clear passed
newoe-20-ipv6 failed east:output-different road:output-different
newoe-21-liveness-clear failed east:EXPECTATION,output-different road:output-different
newoe-22-nat-poc-cop passed
newoe-23-reverse-nat passed east:EXPECTATION road:EXPECTATION
newoe-25-cat-1 failed road:output-different
newoe-25-cat-2 failed road:output-different <== west conns not loaded?
newoe-25-cat-3-4-way failed east:output-different north:output-different road:output-different west:output-different <=== road diagram changed
newoe-25-cat-4 failed east:output-different road:output-different west:output-different <== not clear
newoe-25-cat-5 failed east:output-different road:output-different <=== not clear
dpd-01 failed west:output-different <== timing of SA?
dpd-02 passed
dpd-03 passed
dpd-04 failed east:output-different west:output-different <== address and port changes?
dpd-05 failed west:output-different <== port change
dpd-06 failed west:output-different <== new bad iptables rules?
dpd-07 passed
ikev2-liveness-05 failed east:output-different west:output-different <== no longer KLIPS?
ikev2-liveness-06 failed east:EXPECTATION,output-different road:output-different <== no longer KLIPS?
ikev2-liveness-07 failed east:EXPECTATION,output-different road:output-different <== no longer KLIPS?
ikev2-liveness-08-drop failed east:output-different west:output-different <== no longer KLIPS?
delete-sa-01 passed
delete-sa-03 failed east:output-different west:output-different <== refhim
x509-pluto-01 passed
x509-pluto-02 failed north:output-different <== no outBytes
x509-pluto-03 passed
x509-pluto-04 incomplete east:output-unchecked west:output-unchecked
x509-pluto-frag-00 passed
x509-pluto-frag-01 failed road:output-different <= retransmission message
x509-pluto-frag-02 passed
x509-pluto-frag-03 passed
x509-pluto-frag-04 passed
nat-pluto-01 passed
nat-pluto-02 passed
nat-pluto-03 passed
nat-pluto-04 failed east:output-different road:output-different <== ?
nat-pluto-07 passed
nat-pluto-08 passed
nat-pluto-09 passed
nat-pluto-10 passed east:output-unchecked north:output-unchecked
ikev1-algo-ike-aes-01 passed
ikev1-algo-ike-aes-02 failed east:output-different west:output-different <== no longer KLIPS
basic-pluto-06 passed
psk-pluto-01 passed
psk-pluto-02 incomplete east:output-different road:output-different <== INVALID_HASH_INFORMATION ?
xauth-pluto-03 passed
xauth-pluto-04 passed
xauth-pluto-05 passed
xauth-pluto-05-netkey failed road:output-different
xauth-pluto-06 passed
xauth-pluto-07 passed
xauth-pluto-08 passed
xauth-pluto-09 passed
xauth-pluto-10 passed east:output-unchecked road:output-unchecked
xauth-pluto-11 incomplete east:output-unchecked road:output-unchecked
xauth-pluto-12 failed east:output-different road:output-different <== no longer KLIPS
xauth-pluto-13 passed
xauth-pluto-14 passed
xauth-pluto-15 failed east:output-different road:output-different <== no longer KLIPS
xauth-pluto-16 passed
xauth-pluto-17 failed east:output-different <== swapped IP in XFRM ?
xauth-pluto-18 failed east:output-different <== swapped IP in XFRM ?
xauth-pluto-19 passed
xauth-pluto-20-pam passed
aggr-pluto-01 passed
aggr-pluto-02 passed
ikev1-algo-ike-pfs-01 passed
ikev1-algo-esp-null-01 passed
basic-pluto-01-valgrind passed east:output-unchecked west:output-unchecked
ikev1-algo-ike-bogus-01 passed
basic-pluto-12 failed east:output-different west:output-different <== lots of additional activity
basic-pluto-13-netkey-route passed
pluto-rekey-02 passed
psk-pluto-03 passed
psk-pluto-04 passed
psk-pluto-05 passed
psk-pluto-06 passed
netkey-algo-twofish-01 passed
netkey-algo-serpent-01 passed
netkey-algo-camellia-01 passed
netkey-algo-cast-01 passed
netkey-algo-null-01 passed
netkey-algo-null-02 passed
algo-pluto-01 passed
algo-pluto-04 passed east:output-unchecked west:output-unchecked
algo-pluto-07 passed
algo-pluto-08 passed
algo-pluto-10 passed east:output-whitespace west:output-whitespace
algo-pluto-11 passed
algo-pluto-12-aes-default passed
algo-pluto-13-invalid-3des passed
algo-pluto-14-aes_ctr passed
nat-transport-02 passed
ikev2-21-mode-mismatch-01 failed west:EXPECTATION,output-different
ikev2-21-mode-mismatch-02 passed
ah-pluto-01 passed
ah-pluto-02 passed
ah-pluto-03 passed
ah-pluto-04 passed
ah-pluto-05 passed
ah-pluto-07-klips-netkey passed
ah-pluto-09 passed
ah-pluto-10 passed
ah-pluto-11 passed
interop-ikev1-strongswan-10-ah-initiator-sha256 passed
interop-ikev1-strongswan-11-ah-initiator-sha512 passed
interop-ikev1-strongswan-12-esp-sha2_256 passed
interop-ikev1-strongswan-13-esp-sha2_512 passed
interop-ikev1-strongswan-14-camellia passed
interop-ikev2-strongswan-16-ah-initiator-sha512 failed west:output-different <== cert req changed?
interop-ikev2-strongswan-17-ah-initiator-sha256 failed west:output-different <== cert req changed?
interop-ikev2-strongswan-22-cp-responder-psk passed
interop-ikev2-strongswan-23-initiator-cp passed
interop-ikev2-strongswan-24-aes_ctr passed
interop-ikev2-strongswan-25-ke-mismatch-initiator passed
interop-ikev2-strongswan-26-ke-mismatch-responder failed west:output-different
interop-ikev1-strongswan-15-twofish failed east:output-different west:output-different <== message ignored because it contains an unexpected payload type (ISAKMP_NEXT_HASH)
interop-ikev1-strongswan-16-serpent failed east:output-different west:output-different <== message ignored because it contains an unexpected payload type (ISAKMP_NEXT_HASH)
netkey-klips-pluto-03 passed
klips-netkey-pluto-06 passed
netkey-psk-pluto-06 passed east:output-unchecked road:output-unchecked
compress-pluto-01 passed
compress-pluto-02 passed
compress-pluto-netkey-03 passed
compress-pluto-netkey-klips-04 passed
ipv6-tunnel-mode-02-netkey-netkey failed east:output-different west:output-different <== failure to communicate
ipv6-tunnel-mode-04-rw failed east:output-different road:output-different <== failure to communicate
interop-ikev2-racoon-01-noconn passed east:output-unchecked west:output-unchecked
interop-ikev2-racoon-02-psk-responder failed east:output-different west:output-different <== ?
interop-ikev2-racoon-03-psk-initiator passed east:output-unchecked west:output-unchecked
interop-ikev2-racoon-05-x509-initiator passed east:output-unchecked west:output-unchecked
interop-ikev1-strongswan-02-psk-responder passed
interop-ikev1-strongswan-03-psk-initiator passed
interop-ikev1-strongswan-04-psk-aes-gcm passed
interop-ikev1-strongswan-04-psk-aes-ccm passed
interop-ikev1-strongswan-07-cast128 passed
interop-ikev1-strongswan-08-strongswan-cast passed
interop-ikev2-strongswan-02-psk-responder passed
interop-ikev2-strongswan-03-psk-initiator failed west:output-different <== cert request different [but PSK??]
interop-ikev2-strongswan-04-x509-responder failed west:output-different <== failed to negoriate
interop-ikev2-strongswan-05-psk-aes passed
interop-ikev2-strongswan-05-psk-md5 passed
interop-ikev2-strongswan-06-aes192 passed
interop-ikev2-strongswan-07-strongswan failed west:output-different <== cert request different
interop-ikev2-strongswan-09-psk-aes-ccm passed
interop-ikev2-strongswan-09-psk-aes-gcm passed
interop-ikev2-strongswan-10-nat-initiator passed
interop-ikev2-strongswan-11-nat-initiator passed
interop-ikev2-strongswan-12-nat-responder passed
interop-ikev2-strongswan-13-ah-initiator failed west:output-different <== cert request different
interop-ikev2-strongswan-14-delete-sa failed west:output-different <== cert request different
interop-ikev2-strongswan-15-create_child_sa incomplete east:output-different west:output-different <== ss cannot parse ls message
interop-ikev2-strongswan-17-delete-sa-responder failed west:output-different <== cert request different
interop-ikev2-strongswan-18-psk-cast passed
interop-ikev2-strongswan-19-x509-res-certreq passed
interop-ikev2-strongswan-20-psk-aes_xcbc passed
interop-ikev2-strongswan-27-fragmentation failed west:output-different <== negotiation failed
interop-ikev2-strongswan-28-reauth failed west:output-different <== cert request different
dnssec-pluto-01 failed west:output-different <== westnet-eastnet-etc-hosts-auto-add missing
replay-authip-01 passed
l2tp-01 failed east:output-different north:output-different <== test changed?  No longer KLIPS?
l2tp-02 failed east:output-different north:output-different <== test changed?  No longer KLIPS?
l2tp-02-netkey passed
listen-change-01 passed
fips-default-ikev1-01-nofips-east passed
fips-default-ikev2-01-nofips-east passed
fips-default-ikev1-02-nofips-west passed
fips-default-ikev2-02-nofips-west passed
fips-03-ikev1-md5 passed
fips-04-ikev2-md5 passed
fips-06-ikev1-3des-sha1 passed
fips-07-ikev2-3des-sha256 passed
fips-08-ikev2-x509 failed east:output-different west:output-different <== test changed?  No longer KLIPS?
fips-09-ikev2-gcm passed
netkey-passthrough-ipxfrm passed
netkey-passthrough-00 passed
netkey-passthrough-01 passed
netkey-passthrough-02 passed
netkey-passthrough-03 passed
ikev1-isakmp-reserved-flags-01 passed
ikev1-isakmp-reserved-flags-02 passed
nss-cert-01 passed
nss-cert-01-ikev2 passed
nss-cert-02 passed
nss-cert-02-eku passed
nss-cert-02-eku-combined passed
nss-cert-02-ikev2 passed
nss-cert-03 passed
nss-cert-03-ikev2 passed
nss-cert-04 failed west:output-different <== ignoring retry?
nss-cert-04-ikev2 passed
nss-cert-05 failed east:EXPECTATION west:output-different <== ignoring retry?
nss-cert-05-ikev2 passed
nss-cert-06 passed
nss-cert-07 passed
nss-cert-08-mismatch failed west:EXPECTATION,output-different <== logging seems to have changed
nss-cert-chain-01 passed
nss-cert-chain-02 failed west:output-different <== retransmission different
nss-cert-chain-03 passed
nss-cert-crl-01 passed
nss-cert-crl-01-strict passed
nss-cert-crl-02 failed west:output-different <== duplicate packet
nss-cert-crl-02-fetch passed
nss-cert-crl-03 passed
nss-cert-crl-03-strict failed west:output-different
nss-cert-nosecret failed east:output-different west:output-different <== certs or keys different?
nss-cert-ocsp-01 passed
nss-cert-ocsp-01-chain passed
nss-cert-ocsp-01-ikev2 passed
nss-cert-ocsp-01-strict failed west:output-different <== negotiation failed
nss-cert-ocsp-02 failed east:output-different west:output-different <== no "certificate revoked"; cert accepted!!
nss-cert-ocsp-02-ikev2 failed east:output-different west:output-different <== no "certificate revoked"; cert accepted!!
nss-cert-ocsp-02-strict failed east:output-different west:output-different <== no "certificate revoked"; cert rejected for a different reason
nss-cert-ocsp-03 passed
nss-cert-ocsp-03-ikev2 passed
nss-cert-ocsp-03-strict failed west:output-different <== lots more retransmissions
nss-cert-ocsp-04 failed west:output-different <== negotiation succeeded (mistake??)
nss-cert-ocsp-04-strict failed west:output-different <= old: INVALID_ID_INFORMATION, new: INVALID_KEY_INFORMATION ???
nss-cert-ocsp-05 passed
nss-cert-ocsp-05-ikev2 passed
nss-cert-ocsp-05-strict failed west:output-different <== negotiation failure when success is expected
nss-cert-ocsp-06 failed west:output-different <== negotiation success when failure expected
nss-cert-ocsp-07-nourl failed east:output-different west:output-different <== no "certificate revoked!", negotiation unexpectedly succeeded
ipsec-hostkey-ckaid-01 passed
ipsec-hostkey-ckaid-02-fips passed
ipsec-hostkey-ckaid-03-nsspw passed