[Swan-dev] having failing testscripts fail tests

Andrew Cagney andrew.cagney at gmail.com
Tue Mar 24 16:13:00 EET 2015


One thing I've noticed with tests is that they can end up spending a lot of
time trying to push a train wreck up hill.

For instance, looking in basic-pluto-01/westinit.sh:

/testing/guestbin/swan-prep
# confirm that the network is alive
ping -n -c 4 -I 192.0.1.254 192.0.2.254
# make sure that clear text does not get through
iptables -A INPUT -i eth1 -s 192.0.2.0/24 -j LOGDROP
# confirm with a ping
ping -n -c 4 -I 192.0.1.254 192.0.2.254
ipsec setup start
/testing/pluto/bin/wait-until-pluto-started
ipsec auto --add westnet-eastnet
ipsec auto --status
echo "initdone"

should either swan-prep or wait-until-pluto-started fail then there really
is no point in continuing with the test; yet we do.

The obvious fix is to tweak the code feeding these commands to the remote
end so that it checks the exit code.  Something like:

               test_domain.domain.log("[%s]# %s", basename, line)
                status = test_domain.shell.run(line, timeout=TEST_TIMEOUT)
                if status:
                    # XXX: Can't abort as some ping commands are
                    # expected to fail.
                    test_domain.domain.log("command '%s' failed with status
%d", line, status)

as the comment notes, there are some got-yas:

- some commands are expected to fail; we'd need to tweak them (for instance
changing some "ping ..." commands to  "! ping ...")

- sometimes things just fail (again pings), yet the test stumbles on and
the rest seems to pass; a retry should flush this out

thoughts?

Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20150324/2fa322fe/attachment.html>


More information about the Swan-dev mailing list