[Swan-dev] test suite stability

D. Hugh Redelmeier hugh at mimosa.com
Sun Nov 13 20:44:04 UTC 2016


I ran the test suite twice, once before and once after my latest change.  
As far as I know, my change has no observable effect.  But the two runs 
differ.  I blame instability of the tests, something that needs to be 
fixed (but may be hard to do).

A lot of differences are from IKE retransmissions.  I wonder why?
Some were in the first run and not the second, and some are in the
second and not the first.

Some XFRM listings are different for some reason.

dynamic-iface-01 had a problem with interfaces.

strongswan test results are a bit messy.

< testing/pluto/ikev2-11-simple-psk passed
> testing/pluto/ikev2-11-simple-psk failed west:output-different
A couple of retransmissions of IKE payloads:
 134 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
+010 "westnet-eastnet-ipv4-psk-ikev2"[1] 192.1.2.23 #2: STATE_PARENT_I2: retransmission; will wait 1000ms for response

================
< testing/pluto/ikev2-algo-03-aes-ccm failed west:output-different
> testing/pluto/ikev2-algo-03-aes-ccm passed

One IKE retransmission ELIMINTATED

+010 "westnet-eastnet-ipv4-psk-ikev2-ccm-a" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response

================
< testing/pluto/ikev2-algo-04-aes-gcm256 passed
> testing/pluto/ikev2-algo-04-aes-gcm256 failed west:output-different

+010 "westnet-eastnet-ipv4-psk-ikev2-gcm-c" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response

================
< testing/pluto/ikev2-algo-06-aes-aes_xcbc passed
> testing/pluto/ikev2-algo-06-aes-aes_xcbc failed west:output-different
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response

================
< testing/pluto/netkey-algo-aes_gcm-03 passed
> testing/pluto/netkey-algo-aes_gcm-03 failed west:output-different

Another retransmission.  Also, a packet received whie doing asynch work.

 117 "westnet-eastnet-gcm" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-gcm" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-gcm" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1

================
< testing/pluto/ikev2-algo-ike-sha2-02 failed west:output-different
> testing/pluto/ikev2-algo-ike-sha2-02 passed

 134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_128 integ=sha256_128 prf=sha2_256 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 1000ms for response

================
< testing/pluto/netkey-tfc-03 passed
> testing/pluto/netkey-tfc-03 failed west:output-different

 134 "tfc" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "tfc" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response


================
< testing/pluto/dynamic-iface-01 passed
> testing/pluto/dynamic-iface-01 failed west:output-different
????
-002 adding interface eth1/eth1 192.1.2.66:500
-002 adding interface eth1/eth1 192.1.2.66:4500
-003 two interfaces match "west-float" (eth1, eth1)
-002 "west-float": terminating SAs using this connection

================
< testing/pluto/newoe-20-ipv6 failed east:output-different
> testing/pluto/newoe-20-ipv6 failed east:output-different road:output-different

- a ping packet dropped

-006 #2: "private-or-clear#2001:db8:1:2::/64"[1] ...2001:db8:1:2::23, type=ESP, add_time=1234567890, inBytes=728, outBytes=728, id='ID_NULL'
+006 #2: "private-or-clear#2001:db8:1:2::/64"[1] ...2001:db8:1:2::23, type=ESP, add_time=1234567890, inBytes=624, outBytes=624, id='ID_NULL'

================
< testing/pluto/ikev2-liveness-01 passed east:EXPECTATION
> testing/pluto/ikev2-liveness-01 failed east:EXPECTATION,output-different west:output-different
 134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response

+|    message ID:  00 00 00 00

================
< testing/pluto/ikev2-liveness-03 passed east:EXPECTATION
> testing/pluto/ikev2-liveness-03 failed east:EXPECTATION,output-different west:output-different
 134 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: sent v2I2, expected v2R2 {auth=IKEv2 cipher=aes_gcm_16_256 integ=n/a prf=sha2_512 group=MODP2048}
+010 "westnet-eastnet-ipv4-psk-ikev2" #2: STATE_PARENT_I2: retransmission; will wait 500ms for response

 east #
  ipsec whack --trafficstatus
 000  
+006 #2: "westnet-eastnet-ipv4-psk-ikev2", type=ESP, add_time=1234567890, inBytes=336, outBytes=336, id='@west'
+000  
 east #
  # can be seen on east logs
 east #
  hostname | grep west > /dev/null || grep "IKEv2 liveness action:" /tmp/pluto.log
-"westnet-eastnet-ipv4-psk-ikev2" #2: IKEv2 liveness action: Clearing Connection westnet-eastnet-ipv4-psk-ikev2[0] CK_PERMANENT
 east #
 east #
  if [ -n "`ls /tmp/core* 2>/dev/null`" ]; then echo CORE FOUND; mv /tmp/core* OUTPUT/; fi

================
< testing/pluto/ikev1-algo-ike-aes-02 failed west:output-different
> testing/pluto/ikev1-algo-ike-aes-02 passed

 117 "westnet-eastnet-3des" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-3des" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-3des" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1

================
< testing/pluto/xauth-pluto-05 failed road:output-different
> testing/pluto/xauth-pluto-05 passed

 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 

================
< testing/pluto/xauth-pluto-06 failed road:output-different
> testing/pluto/xauth-pluto-06 passed
 	enc cbc(aes) 0xENCKEY
 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
================
< testing/pluto/xauth-pluto-07 failed road:output-different
> testing/pluto/xauth-pluto-07 passed

 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
================
< testing/pluto/xauth-pluto-08 passed
> testing/pluto/xauth-pluto-08 failed road:output-different
 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 

================
< testing/pluto/xauth-pluto-12 passed
> testing/pluto/xauth-pluto-12 failed road:output-different
 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 

================
< testing/pluto/xauth-pluto-14 failed road:output-different
> testing/pluto/xauth-pluto-14 passed

 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
================
< testing/pluto/xauth-pluto-18 failed east:output-different road:output-different
> testing/pluto/xauth-pluto-18 failed east:output-different

 		proto esp reqid REQID mode tunnel
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
================
< testing/pluto/ikev1-algo-esp-null-01 passed
> testing/pluto/ikev1-algo-esp-null-01 failed west:output-different

 117 "westnet-eastnet-null" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-null" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-null" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/interop-ikev2-strongswan-22-cp-responder-psk failed east:output-different road:output-different
> testing/pluto/interop-ikev2-strongswan-22-cp-responder-psk failed road:output-different

older:

 Security Associations (1 up, 0 connecting):
-roadnet-eastnet-ikev2[2]: ESTABLISHED XXX seconds ago, 192.1.2.23[east]...192.1.3.209[road]
+roadnet-eastnet-ikev2[2]: ESTABLISHED XXX second ago, 192.1.2.23[east]...192.1.3.209[road]

+000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
 000 algorithm AH/ESP auth: id=250, name=AUTH_ALGORITHM_AES_CMAC_96, keysizemin=128, keysizemax=128
-000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128


newer:

+000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128
 000 algorithm AH/ESP auth: id=250, name=AUTH_ALGORITHM_AES_CMAC_96, keysizemin=128, keysizemax=128
-000 algorithm AH/ESP auth: id=9, name=AUTH_ALGORITHM_AES_XCBC, keysizemin=128, keysizemax=128

 XFRM policy:
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket out priority 0 ptype main 
+src 0.0.0.0/0 dst 0.0.0.0/0 
+	socket in priority 0 ptype main 

================
< testing/pluto/compress-pluto-netkey-03 passed
> testing/pluto/compress-pluto-netkey-03 failed west:output-different

 117 "westnet-eastnet-compress" #2: STATE_QUICK_I1: initiate
+010 "westnet-eastnet-compress" #2: STATE_QUICK_I1: retransmission; will wait 500ms for response
+002 "westnet-eastnet-compress" #2: discarding packet received during asynchronous work (DNS or crypto) in STATE_QUICK_I1
================
< testing/pluto/interop-ikev2-strongswan-10-nat-initiator passed
> testing/pluto/interop-ikev2-strongswan-10-nat-initiator failed east:output-different road:output-different

 XFRM state:
-src 192.1.2.254 dst 192.1.2.23
-	proto esp spi 0xSPISPIXX reqid REQID mode tunnel
-	replay-window 32 flag af-unspec
-	auth-trunc hmac(sha512) 0xHASHKEY 256
-	enc cbc(aes) 0xENCKEY
-	encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
-src 192.1.2.23 dst 192.1.2.254
-	proto esp spi 0xSPISPIXX reqid REQID mode tunnel
-	replay-window 32 flag af-unspec
-	auth-trunc hmac(sha512) 0xHASHKEY 256
-	enc cbc(aes) 0xENCKEY
-	encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
 XFRM policy:
-src 192.0.2.0/24 dst 192.0.4.0/24 
-	dir out priority 2344 ptype main 
-	tmpl src 192.1.2.23 dst 192.1.2.254
-		proto esp reqid REQID mode tunnel
-src 192.0.4.0/24 dst 192.0.2.0/24 
-	dir fwd priority 2344 ptype main 
-	tmpl src 192.1.2.254 dst 192.1.2.23
-		proto esp reqid REQID mode tunnel
-src 192.0.4.0/24 dst 192.0.2.0/24 
-	dir in priority 2344 ptype main 
-	tmpl src 192.1.2.254 dst 192.1.2.23
-		proto esp reqid REQID mode tunnel

Lots bad on Road side, including:

-Security Associations (1 up, 0 connecting):
-road-eastnet-ikev2[1]: ESTABLISHED XXX seconds ago, 192.1.3.209[road]...192.1.2.23[east]
-road-eastnet-ikev2{1}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: SPISPI_i SPISPI_o
-road-eastnet-ikev2{1}:   192.0.4.0/24 === 192.0.2.0/24
+Security Associations (0 up, 0 connecting):
+  none

================
< testing/pluto/interop-ikev2-strongswan-13-ah-initiator passed
> testing/pluto/interop-ikev2-strongswan-13-ah-initiator failed east:output-different west:output-different
 east NOW
 XFRM state:
-src 192.1.2.45 dst 192.1.2.23
-	proto ah spi 0xSPISPIXX reqid REQID mode tunnel
-	replay-window 32 flag af-unspec
-	auth-trunc hmac(sha1) 0xHASHKEY 96
-src 192.1.2.23 dst 192.1.2.45
-	proto ah spi 0xSPISPIXX reqid REQID mode tunnel
-	replay-window 32 flag af-unspec
-	auth-trunc hmac(sha1) 0xHASHKEY 96
 XFRM policy:
-src 192.0.2.0/24 dst 192.0.1.0/24 
-	dir out priority 2344 ptype main 
-	tmpl src 192.1.2.23 dst 192.1.2.45
-		proto ah reqid REQID mode tunnel
-src 192.0.1.0/24 dst 192.0.2.0/24 
-	dir fwd priority 2344 ptype main 
-	tmpl src 192.1.2.45 dst 192.1.2.23
-		proto ah reqid REQID mode tunnel
-src 192.0.1.0/24 dst 192.0.2.0/24 
-	dir in priority 2344 ptype main 
-	tmpl src 192.1.2.45 dst 192.1.2.23
-		proto ah reqid REQID mode tunnel

lots on west, including:

 west #
  if [ -f /var/run/charon.pid ]; then strongswan status ; fi
-Security Associations (1 up, 0 connecting):
-westnet-eastnet-ikev2[1]: ESTABLISHED XXX seconds ago, 192.1.2.45[west]...192.1.2.23[east]
-westnet-eastnet-ikev2{1}:  INSTALLED, TUNNEL, reqid 1, AH SPIs: SPISPI_i SPISPI_o
-westnet-eastnet-ikev2{1}:   192.0.1.0/24 === 192.0.2.0/24
+Security Associations (0 up, 0 connecting):
+  none
 west #


================
< testing/pluto/interop-ikev2-strongswan-21-transport-03 failed west:output-different
> testing/pluto/interop-ikev2-strongswan-21-transport-03 failed east:output-different west:output-different

I got tired of analysis.

================
< testing/pluto/fips-06-ikev1-3des-sha1 failed west:output-different
> testing/pluto/fips-06-ikev1-3des-sha1 passed

I got tired of analysis.

================
< testing/pluto/netkey-passthrough-ipxfrm unresolved east:output-missing west:output-missing
> testing/pluto/netkey-passthrough-ipxfrm passed

I got tired of analysis.

================


More information about the Swan-dev mailing list