[Swan-dev] speed of tests on three systems

[Antony Antony]

On Mon, Oct 22, 2018 at 10:03:56AM -0400, Andrew Cagney wrote:
> There are graphs at the bottom of
> https://libreswan.org/wiki/Test_Suite that should help
> 
> The model is roughly:
> 
>     boot-workers | test-runners
> 
> How many tests did you run in parallel?  Per above graphs, testing in
> parallel has a far bigger effect then booting in parallel.  This is
> because time is wasted in two places:
> 
> - each boot wastes seconds bringing up the network
> for your machine look at the time it takes for the build machine to
> get to the login prompt (~1s) vs the time it takes for a test machine
> to get to the login prompt (>>1s) (look for login: in output)
> The big benefit of this is being able to reduce the #parallel-boots
> (aka kvm-workers) which should free up cpu for running more tests in
> parallel
> 
> - (as antony point) the real time-sink is all the sleeps (we can be
> talking minutes wasted) embedded in our tests

well, I pointed out when I realized you are running in serial:). I am hopping the concurrency can help speed up instead of banning tests with sleeps. IMHO we need both. May be the scripts can be bit smarter based on history? 
I urn with multiple test runners equivalents, using swantest. As an indication my runs take around 4:30 or so - may be a big difference is using tmpfs for qcow2 files.

In the long term I hope to have both long running tests and short tests.
I imagine once the test is setup KVM do not demand much resources. So test runners could be smart about long running tests, especially with tmpfs and sufficient ram. Assuming host do not swap a lot.
 
> Locally I use one boot-worker and two parallel tests (but I might be
> testing 3 trees at once).
> 
> On Sun, 21 Oct 2018 at 11:12, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> >
> > Last night I ran the tests on three different machines.  The tests may
> > have been with slightly different versions of the tree, but all were fresh
> > within the last day. 



> >
> > Each machine used KVM_WORKERS=2
> > Each machine is running Fedora 24 for x86-64, with recent updates.
> > Each machine was only running the test suite.
> 
> > redtiny was running the test suite from the Gnome Desktop; redox and
> > redbird ran it from an SSH session.  I don't imagine that matters.>
> > redtiny: 5:46:40.231926 real time
> > Lenovo ThinkCentre M93p Tiny,
> > i5-4570T CPU @ 2.90GHz, 4 core,
> > 16G RAM (two sticks)
> > laptop HDD 7200RPM


Hugh, 

thanks for sharing the numbers.

I am curious to see how fast it would be if you used tmpfs. 
and possibly more "test-runners". 

KVM_PREFIX= t1. t2. t3. t4.
KVM_WORKERS=2
KVM_POOLDIR=/tmp/swanpool

to speed up the installation. put link to fedora iso in the directory.
Fedora-Server-dvd-x86_64-28-1.1.iso -> /home/build/pool/Fedora-Server-dvd-x86_64-28-1.1.iso

Otherwise the install script will download F28 iso every time you rebuild after a reboot.

NOTE: you have to increase your host's tmpfs size. You may need about 10GB ramfs. A 32GB RAM host is ideal for this:)

Or/and decrease VIRT_DISK_SIZE_GB=6GB or so. 

-antony