[Swan-dev] speed of tests on three systems

Andrew Cagney andrew.cagney at gmail.com
Mon Oct 22 14:03:56 UTC 2018

There are graphs at the bottom of
https://libreswan.org/wiki/Test_Suite that should help

The model is roughly:

    boot-workers | test-runners

How many tests did you run in parallel?  Per above graphs, testing in
parallel has a far bigger effect then booting in parallel.  This is
because time is wasted in two places:

- each boot wastes seconds bringing up the network
for your machine look at the time it takes for the build machine to
get to the login prompt (~1s) vs the time it takes for a test machine
to get to the login prompt (>>1s) (look for login: in output)
The big benefit of this is being able to reduce the #parallel-boots
(aka kvm-workers) which should free up cpu for running more tests in

- (as antony point) the real time-sink is all the sleeps (we can be
talking minutes wasted) embedded in our tests

Locally I use one boot-worker and two parallel tests (but I might be
testing 3 trees at once).

On Sun, 21 Oct 2018 at 11:12, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> Last night I ran the tests on three different machines.  The tests may
> have been with slightly different versions of the tree, but all were fresh
> within the last day.
> Each machine used KVM_WORKERS=2
> Each machine is running Fedora 24 for x86-64, with recent updates.
> Each machine was only running the test suite.

> redtiny was running the test suite from the Gnome Desktop; redox and
> redbird ran it from an SSH session.  I don't imagine that matters.>
> redtiny: 5:46:40.231926 real time
> Lenovo ThinkCentre M93p Tiny,
> i5-4570T CPU @ 2.90GHz, 4 core,
> 16G RAM (two sticks)
> laptop HDD 7200RPM
> redox: 5:23:38.344347 real time
> Lenovo ThinkCentre M93p Tiny,
> i5-4570T CPU @ 2.90GHz, 4 core,
> 8G RAM (one stick)
> redbird: 5:23:12.254131 real time
> Dell Optiplex 990 (A24 firmware with spectre/meltdown slowdowns)
> i5-2400 CPU @ 3.10GHz,  4 core,
> 12G RAM (two unmatched sticks)
> All times seemed quite similar.  Each machine was distinct in a couple
> of dimensions.  It is possible that differences cancelled out.  But it
> seems more likely that the differences didn't affect the time very
> much.
> It looks as if the following don't have severe effects on run time:
> - RAM beyond 8G (with two workers)
> - processor speed (but the processors were fairly close in speed)
> The system with the HDD was only 7% slower (perhaps the disk speed
> difference was counteracted by greater RAM).
> These results don't tell us where all the time goes.

More information about the Swan-dev mailing list