[Swan-dev] test suite stability
D. Hugh Redelmeier
hugh at mimosa.com
Sun Nov 13 20:44:04 UTC 2016
I ran the test suite twice, once before and once after my latest change.
As far as I know, my change has no observable effect. But the two runs
differ. I blame instability of the tests, something that needs to be
fixed (but may be hard to do).
A lot of differences are from IKE retransmissions. I wonder why?
Some were in the first run and not the second, and some are in the
second and not the first.
Some XFRM listings are different for some reason.
dynamic-iface-01 had a problem with interfaces.
strongswan test results are a bit messy.
< testing/pluto/ikev2-11-simple-psk passed
> testing/pluto/ikev2-11-simple-psk failed west:output-different
A couple of retransmissions of IKE payloads:
134 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
+010 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: retransmission; will wait 1000ms for response
================
< testing/pluto/ikev2-algo-03-aes-ccm failed west:output-different
> testing/pluto/ikev2-algo-03-aes-ccm passed
One IKE retransmission ELIMINTATED
+010 "westnet-eastnet-ipv4-psk-ikev2-ccm-a" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
================
< testing/pluto/ikev2-algo-04-aes-gcm256 passed
> testing/pluto/ikev2-algo-04-aes-gcm256 failed west:output-different
+010 "westnet-eastnet-ipv4-psk-ikev2-gcm-c" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
================
< testing/pluto/ikev2-algo-06-aes-aes_xcbc passed
> testing/pluto/ikev2-algo-06-aes-aes_xcbc failed west:output-different
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
================
< testing/pluto/netkey-algo-aes_gcm-03 passed
> testing/pluto/netkey-algo-aes_gcm-03 failed west:output-different
Another retransmission. Also, a packet received whie doing asynch work.
117 "westnet-eastnet-gcm" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-gcm" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-gcm" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/ikev2-algo-ike-sha2-02 failed west:output-different
> testing/pluto/ikev2-algo-ike-sha2-02 passed
134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_128 integ=sha256_128 prf=sha2_256 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 1000ms for response
================
< testing/pluto/netkey-tfc-03 passed
> testing/pluto/netkey-tfc-03 failed west:output-different
134 "tfc" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "tfc" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
================
< testing/pluto/dynamic-iface-01 passed
> testing/pluto/dynamic-iface-01 failed west:output-different
????
-002 adding interface eth1/eth1 192.1.2.66:500
-002 adding interface eth1/eth1 192.1.2.66:4500
-003 two interfaces match "west-float" (eth1, eth1)
-002 "west-float": terminating SAs using this connection
================
< testing/pluto/newoe-20-ipv6 failed east:output-different
> testing/pluto/newoe-20-ipv6 failed east:output-different road:output-different
- a ping packet dropped
-006 #2: "private-or-clear#2001:db8:1:2::/64"[1] ...2001:db8:1:2::23, type=ESP, add_time=1234567890, inBytes=728, outBytes=728, id='ID_NULL'
+006 #2: "private-or-clear#2001:db8:1:2::/64"[1] ...2001:db8:1:2::23, type=ESP, add_time=1234567890, inBytes=624, outBytes=624, id='ID_NULL'
================
< testing/pluto/ikev2-liveness-01 passed east:EXPECTATION
> testing/pluto/ikev2-liveness-01 failed east:EXPECTATION,output-different west:output-different
134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
+| message ID: 00 00 00 00
================
< testing/pluto/ikev2-liveness-03 passed east:EXPECTATION
> testing/pluto/ikev2-liveness-03 failed east:EXPECTATION,output-different west:output-different
134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
east #
ipsec whack --trafficstatus
000
+006 #2: "westnet-eastnet-ipv4-psk-ikev2", type=ESP, add_time=1234567890, inBytes=336, outBytes=336, id='@west'
+000
east #
# can be seen on east logs
east #
hostname | grep west > /dev/null || grep "IKEv2 liveness action:" /tmp/pluto.log
-"westnet-eastnet-ipv4-psk-ikev2" #2: IKEv2 liveness action: Clearing Connection westnet-eastnet-ipv4-psk-ikev2[0] CK_PERMANENT
east #
east #
if [ -n "`ls /tmp/core* 2>/dev/null`" ]; then echo CORE FOUND; mv /tmp/core* OUTPUT/; fi
================
< testing/pluto/ikev1-algo-ike-aes-02 failed west:output-different
> testing/pluto/ikev1-algo-ike-aes-02 passed
117 "westnet-eastnet-3des" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-3des" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-3des" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/xauth-pluto-05 failed road:output-different
> testing/pluto/xauth-pluto-05 passed
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-06 failed road:output-different
> testing/pluto/xauth-pluto-06 passed
enc cbc(aes) 0xENCKEY
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-07 failed road:output-different
> testing/pluto/xauth-pluto-07 passed
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-08 passed
> testing/pluto/xauth-pluto-08 failed road:output-different
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-12 passed
> testing/pluto/xauth-pluto-12 failed road:output-different
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-14 failed road:output-different
> testing/pluto/xauth-pluto-14 passed
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/xauth-pluto-18 failed east:output-different road:output-different
> testing/pluto/xauth-pluto-18 failed east:output-different
proto esp reqid REQID mode tunnel
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/ikev1-algo-esp-null-01 passed
> testing/pluto/ikev1-algo-esp-null-01 failed west:output-different
117 "westnet-eastnet-null" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-null" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-null" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/interop-ikev2-strongswan-22-cp-responder-psk failed east:output-different road:output-different
> testing/pluto/interop-ikev2-strongswan-22-cp-responder-psk failed road:output-different
older:
Security Associations (1 up, 0 connecting):
-roadnet-eastnet-ikev2[2]: ESTABLISHED XXX seconds ago, 192.1.2.23[east]...192.1.3.209[road]
+roadnet-eastnet-ikev2[2]: ESTABLISHED XXX second ago, 192.1.2.23[east]...192.1.3.209[road]
+000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
000 algorithm AH/ESP auth: id=250, name=AUTH_ALGORITHM_AES_CMAC_96, keysizemin=128, keysizemax=128
-000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
newer:
+000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
000 algorithm AH/ESP auth: id=250, name=AUTH_ALGORITHM_AES_CMAC_96, keysizemin=128, keysizemax=128
-000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket out priority 0 ptype main
+src 0.0.0.0/0 dst 0.0.0.0/0
+ socket in priority 0 ptype main
================
< testing/pluto/compress-pluto-netkey-03 passed
> testing/pluto/compress-pluto-netkey-03 failed west:output-different
117 "westnet-eastnet-compress" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-compress" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-compress" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/interop-ikev2-strongswan-10-nat-initiator passed
> testing/pluto/interop-ikev2-strongswan-10-nat-initiator failed east:output-different road:output-different
XFRM state:
-src 192.1.2.254 dst 192.1.2.23
- proto esp spi 0xSPISPIXX reqid REQID mode tunnel
- replay-window 32 flag af-unspec
- auth-trunc hmac(sha512) 0xHASHKEY 256
- enc cbc(aes) 0xENCKEY
- encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
-src 192.1.2.23 dst 192.1.2.254
- proto esp spi 0xSPISPIXX reqid REQID mode tunnel
- replay-window 32 flag af-unspec
- auth-trunc hmac(sha512) 0xHASHKEY 256
- enc cbc(aes) 0xENCKEY
- encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
XFRM policy:
-src 192.0.2.0/24 dst 192.0.4.0/24
- dir out priority 2344 ptype main
- tmpl src 192.1.2.23 dst 192.1.2.254
- proto esp reqid REQID mode tunnel
-src 192.0.4.0/24 dst 192.0.2.0/24
- dir fwd priority 2344 ptype main
- tmpl src 192.1.2.254 dst 192.1.2.23
- proto esp reqid REQID mode tunnel
-src 192.0.4.0/24 dst 192.0.2.0/24
- dir in priority 2344 ptype main
- tmpl src 192.1.2.254 dst 192.1.2.23
- proto esp reqid REQID mode tunnel
Lots bad on Road side, including:
-Security Associations (1 up, 0 connecting):
-road-eastnet-ikev2[1]: ESTABLISHED XXX seconds ago, 192.1.3.209[road]...192.1.2.23[east]
-road-eastnet-ikev2{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: SPISPI_i SPISPI_o
-road-eastnet-ikev2{1}: 192.0.4.0/24 === 192.0.2.0/24
+Security Associations (0 up, 0 connecting):
+ none
================
< testing/pluto/interop-ikev2-strongswan-13-ah-initiator passed
> testing/pluto/interop-ikev2-strongswan-13-ah-initiator failed east:output-different west:output-different
east NOW
XFRM state:
-src 192.1.2.45 dst 192.1.2.23
- proto ah spi 0xSPISPIXX reqid REQID mode tunnel
- replay-window 32 flag af-unspec
- auth-trunc hmac(sha1) 0xHASHKEY 96
-src 192.1.2.23 dst 192.1.2.45
- proto ah spi 0xSPISPIXX reqid REQID mode tunnel
- replay-window 32 flag af-unspec
- auth-trunc hmac(sha1) 0xHASHKEY 96
XFRM policy:
-src 192.0.2.0/24 dst 192.0.1.0/24
- dir out priority 2344 ptype main
- tmpl src 192.1.2.23 dst 192.1.2.45
- proto ah reqid REQID mode tunnel
-src 192.0.1.0/24 dst 192.0.2.0/24
- dir fwd priority 2344 ptype main
- tmpl src 192.1.2.45 dst 192.1.2.23
- proto ah reqid REQID mode tunnel
-src 192.0.1.0/24 dst 192.0.2.0/24
- dir in priority 2344 ptype main
- tmpl src 192.1.2.45 dst 192.1.2.23
- proto ah reqid REQID mode tunnel
lots on west, including:
west #
if [ -f /var/run/charon.pid ]; then strongswan status ; fi
-Security Associations (1 up, 0 connecting):
-westnet-eastnet-ikev2[1]: ESTABLISHED XXX seconds ago, 192.1.2.45[west]...192.1.2.23[east]
-westnet-eastnet-ikev2{1}: INSTALLED, TUNNEL, reqid 1, AH SPIs: SPISPI_i SPISPI_o
-westnet-eastnet-ikev2{1}: 192.0.1.0/24 === 192.0.2.0/24
+Security Associations (0 up, 0 connecting):
+ none
west #
================
< testing/pluto/interop-ikev2-strongswan-21-transport-03 failed west:output-different
> testing/pluto/interop-ikev2-strongswan-21-transport-03 failed east:output-different west:output-different
I got tired of analysis.
================
< testing/pluto/fips-06-ikev1-3des-sha1 failed west:output-different
> testing/pluto/fips-06-ikev1-3des-sha1 passed
I got tired of analysis.
================
< testing/pluto/netkey-passthrough-ipxfrm unresolved east:output-missing west:output-missing
> testing/pluto/netkey-passthrough-ipxfrm passed
I got tired of analysis.
================
More information about the Swan-dev
mailing list