[Swan-dev] tests and bugs (might be interesting to GSoC students) Re: Marking never pass tests with WIP

[Paul Wouters]

On Thu, 9 Mar 2017, Andrew Cagney wrote:

Note: These problems listed will have minor test cases issues that are
nice small problems for GSOC students to get familiar with our code
base. But some might also be real and difficult bugs. If you want to
have a go and some of these, do join the #swan channel and ask for help.
If you want to know how to run tests, see:

https://libreswan.org/wiki/Test_Suite


> Looking at http://testing.libreswan.org/results/testing/ and excluding
> WIP we seem to consistently have 100 failing tests (you can eyeball
> this from the graph) so I did a little analysis using the script
> testing/web/never-passed.sh:
>
> - 45 are 'original' and have never ever passed
> - 16 are more recent yet also show no sign of passing

It might be worth redoing the list you made with the PEXPECT removed?

Some comments from the top of my head on your list:

> 0 230 230 ikev2-allow-narrow-01
> 0 230 230 ikev2-allow-narrow-02
> 0 230 230 ikev2-allow-narrow-03

These used to pass, but they are testing failure path. I think the
logging in the path changed.

> 0 230 230 interop-ikev2-racoon-02-psk-responder

We have no good racoon sanitizer.

> 0 230 230 interop-ikev2-strongswan-19-x509-res-certreq
> 0 230 230 interop-ikev2-strongswan-21-transport-02
> 0 230 230 interop-ikev2-strongswan-21-transport-03
> 0 230 230 interop-ikev2-strongswan-26-ke-mismatch-responder
> 0 230 230 interop-ikev2-strongswan-27-fragmentation

Should all pass, but might be kernel/strongswan version specific.

> 0 230 230 newoe-07-ike-replace-initiator-esp
> 0 230 230 newoe-15-portpass
> 0 230 230 newoe-18-poc-block
> 0 230 230 newoe-18-poc-blockall
> 0 230 230 newoe-18-private-clear
> 0 230 230 newoe-18-private-clearall
> 0 230 230 newoe-19-poc-poc-clear
> 0 230 230 newoe-20-ipv6
> 0 230 230 newoe-21-liveness-clear

Should all pass.

> 0 230 230 nss-cert-08-mismatch

Recently got a code fix to get it to pass, but needs updatin.

> 0 230 230 nss-cert-nosecret
> 0 230 230 nss-cert-ocsp-01-strict
> 0 230 230 nss-cert-ocsp-02
> 0 230 230 nss-cert-ocsp-02-ikev2
> 0 230 230 nss-cert-ocsp-03-strict
> 0 230 230 nss-cert-ocsp-04
> 0 230 230 nss-cert-ocsp-05-strict
> 0 230 230 nss-cert-ocsp-06
> 0 230 230 nss-cert-ocsp-07-nourl

Some people show a weird ocspd problem on nic. Some people have
no ocspd on nic :P

> 0 218 218 dnssec-pluto-01

I'm working on this code and will pick it up

> 0 218 218 dpd-01
> 0 218 218 dpd-04
> 0 218 218 dpd-06

the dpd and liveness tests are awful to get consistent :( Usually
all these are false positives.

> 0 218 218 l2tp-01
> 0 218 218 l2tp-02

I've looked at this a few times. It seems a real bug, but I haven't
been able to locate it yet.

> 0 177 177 seccomp-01-enabled
> 0 177 177 seccomp-02-tolerant

experimental code needs fixing

> 0 152 152 nss-cert-09-notyetvalid-initiator
> 0 152 152 nss-cert-09-notyetvalid-initiator-ikev2
> 0 152 152 nss-cert-10-notyetvalid-responder
> 0 152 152 nss-cert-10-notyetvalid-responder-ikev2

These should work, but do require libfaketime and faketime seems to
not work properly when using docker. I thought these should pass
using kvm though.

> 0 152 152 nss-cert-ocsp-08-post

same ocspd issue?

> 0 75 75 addconn-01

A recent add for testing a parser bug. The bug is still there :)

> 0 53 53 certoe-01-whack

should work?

> 0 53 53 certoe-07-nat-2-clients
> 0 53 53 certoe-08-nat-packet-cop-restart
> 0 49 49 nss-cert-chain-01-ikev2
> 0 49 49 nss-cert-chain-03-ikev2
> 0 49 49 nss-cert-chain-04-ikev2

Should also work.

> 0 37 37 ikev1-aggr-sendcert-01

Recently added test case to show a bug where we did not properly
send CERT payloads in Aggressive Mode. I think this bug still needs
fixing. It was fixed recently for Main Mode I think.

> 0 18 18 netkey-vti-06c

I think this one shows a known kernel bug.

> 5 225 230 nflog-01-global

This is a weird one too. I looked at it a few times. It should work.

> 6 224 230 ipsec-hostkey-ckaid-02-fips
> 8 0 8 ikev2-algo-ike-dh-ecp-01
> 8 0 8 interop-ikev2-strongswan-33-ike-dh-ecp-responder

I assume you know best about these? :)

> 33 197 230 nss-cert-ocsp-01
> 33 197 230 nss-cert-ocsp-01-chain
> 33 197 230 nss-cert-ocsp-01-ikev2
> 33 197 230 nss-cert-ocsp-03
> 33 197 230 nss-cert-ocsp-03-ikev2
> 33 197 230 nss-cert-ocsp-05
> 33 197 230 nss-cert-ocsp-05-ikev2

same ocspd?

> 41 6 47 delete-sa-09-rhbz1311360-c32

This shows a bug that is fixed, so it should work. Perhaps test just
needs updating.

> 45 8 53 ikev2-asymmetric-01-parsing

This is a new one to ensure we parse all auth/authby properly. Probably
needs updating.

> 47 1 48 delete-sa-08-rhbz1311360

Also shows a bug that is fixed so should work or might need an update.

> 48 1 49 nss-cert-chain-02-ikev2
> 48 1 49 nss-cert-chain-04

Recent work should have fixed these too.

> 50 3 53 ikev2-asymmetric-04-null-rsaraw
> 50 3 53 ikev2-asymmetric-07-psk-rsaraw
> 50 3 53 ikev2-asymmetric-08-psk-rsacert
> 50 3 53 ikev2-asymmetric-09-rsaraw-null
> 50 3 53 ikev2-asymmetric-12-rsacert-null
> 50 3 53 ikev2-asymmetric-13-rsacert-psk
> 50 3 53 ikev2-asymmetric-15-rsaraw-rsaraw

Should all work. There might be a bug in finding the right
connection in some combinations. I forgot what the current
status was here.

> 63 167 230 ikev2-01-fallback-ikev1

The code tested here is slated to be removed, once we make
conns to be either ikev1 or ikev2 but not both. Similar with
the bidown attack test case. But it should currently work
with the fallback.

> 66 39 105 klips-algo-twofish-01
> 71 34 105 klips-algo-serpent-01

should work?

> 72 100 172 ikev1-impair-gx-02

some of these were the pexect problems that got resolved two
days ago.

> 89 16 105 klips-algo-cast-01

should work ?

> 113 105 218 netkey-audit-01

Requires proper USE_LINUX_AUDIT=true which is not the default.
I think if debian/ubuntu support the auditd now, we can change
the default to be enabled.

> 159 59 218 netkey-algo-twofish-01

should work.

> 160 58 218 netkey-algo-aes_ccm-01
> 160 58 218 netkey-algo-aes_gcm-01
> 160 58 218 netkey-algo-aes_gcm-02
> 160 58 218 netkey-algo-aes_gcm-03

should work?

> 167 63 230 ikev2-delete-02

We changed some of our delete/restart bhaviour, and we are not
done with the changes yet. So some of these delete test cases
might be in flux for a bit.

> 167 63 230 netkey-tfc-01
> 167 63 230 netkey-tfc-02
> 171 59 230 netkey-tfc-03

Requires new enough kernel (4.x maybe?)

> 172 58 230 ikev2-invalid-ke-02-wrong-modp
> 174 56 230 ikev2-11-simple-psk
> 174 56 230 ikev2-algo-ike-sha2-03

should work?

> 174 56 230 xauth-pluto-05
> 174 56 230 xauth-pluto-06
> 174 56 230 xauth-pluto-09
> 174 56 230 xauth-pluto-19
> 175 55 230 basic-pluto-06

these might need fixing.

> 177 53 230 ikev2-delete-01

See above.

> 179 39 218 netkey-algo-null-01
> 179 39 218 netkey-algo-serpent-01
> 180 50 230 ikev2-12-transport-psk

should work.

> 181 49 230 x509-pluto-02
> 181 49 230 x509-pluto-03

might be obsolete ones, will need to look.

> 181 37 218 ikev1-isakmp-reserved-flags-01
> 181 37 218 ikev1-isakmp-reserved-flags-02

I think the failure path changed output.

> 184 46 230 algo-pluto-13-invalid-3des

I think this tested a bogus keysize, but then we
switched to rejecting to load these. Might require
a new impair to send bogus keysize value over the
wire.

> 184 46 230 ikev2-21-mode-mismatch-02

This might have changed. The RFC says if client requests
transport mode, but server refuses transport mode, client
needs to use tunnel mode. We might insist on exact match,
or changed to following the spec without updating the
test case.

> 201 29 230 ikev2-invalid-ke-01-invalid-modp
> 201 29 230 ikev2-invalid-ke-03-default-initiator
> 201 29 230 ikev2-invalid-ke-04-default-responder

Should work but failure path output might have changed?

> 212 18 230 ikev2-isakmp-reserved-flags-01
> 212 18 230 ikev2-isakmp-reserved-flags-02
> 212 18 230 ikev2-minor-version-initiator
> 212 18 230 ikev2-minor-version-responder
> 212 18 230 ikev2-payload-reserved-flags-01

Same.

> 213 17 230 ikev2-algo-esn-01
> 213 17 230 ikev2-algo-esn-03
> 213 17 230 ikev2-algo-esn-05
> 213 17 230 ikev2-algo-esn-06

Might depend on kernel version. Might need 4.x

> 213 5 218 l2tp-02-netkey

A real concern / bug like the other two l2tp test cases.

> 214 4 218 netkey-vti-02

I believe these first two VTI testcases predated some of the
VTI features and need updating. I thought I did that though?

> 229 1 230 newoe-18-*

And all the others. These should pass, but are known to be
sensitive to machine speeds with false positives on whether
it eats 1 or 2 packets.

Paul