[Swan-dev] [Testing] Libreswan Testing Future - Call Notes (from May 16)

[Andrew Cagney]

Thanks for the notes.


> Testing OS
> ----------
> Any (Linux) OS should be supported by the testsuite ideally, this allows
> us to do OS testing. In order to achieve this goal system-specific and
> not widely accepted system features should better be avoided (such as 9P
> for instance). Host testing environment should be stable, should not
> obsolete too fast or be significantly changed very often. I thinks there
> should be at least one distribution on which the testsuite works flawlessly.

Yea, any OS - testing Windows interaction would be nice.

> Testing with KVM
> ----------------
> KVM is now used to emulate network under test. This approach is slow
> because of various reasons (not only related to KVM): slow boot of
> guests, sanity checks (eg. checking that something is not reachable
> means waiting for timeout), transmogrification slow-down, reboots of
> guests between subsequent tests need to clean-up network stacks. KVM
> approach does not scale well, it works fine for 1-3 network nodes but
> fails badly for 100. Parallel test execution does not fit into KVM by
> the same principle. The only non-network shared directory for KVM is
> represented by 9P filesystem which is rather slow, seemingly not
> maintained anymore and missing completely in RHEL. Avoiding 9P means
> using network between host and guests (eg. NFS) which might be fragile
> and potential source of problems. Last but no least, KVM seems to losing
> drive.

Its probably worth clarifying on where the time goes:

- first 20-30 seconds of host-cpu time get sucked up booting each VM;
what ever it is it is clearly "silly"
The suspects, in order, are Fedora and Transmogrify.  KVM is very
unlikely as a suspect, debian, for instance, and on the same H/W,
boots in seconds.
Perhaps we should just switch to Debian.

- then 30-40 seconds is spent looking at the wall clock while various
timeouts expire
There are low hanging fruit here, for instance not waiting 10 seconds
to confirm a "reliable" network is down.  However some of the more
complex cases will involve interaction using an expect like tool.