[Swan-dev] Test Suite: speed effects of hardware differences

[Andrew Cagney]

Here's another data point:

If I run:
    ./kvm recheck
i.e., create $(KVM_PREFIXES) threads (here 3) and then distribute all
the tests, I get:
    kvmrunner 1:45:06.09: final test results:
    kvmrunner 1:45:06.09:   stats/good/passed/previous=unresolved: 3
    kvmrunner 1:45:06.09:   tests/good/results/failed: 9
    kvmrunner 1:45:06.09:   total/passed: 861
which is nearly _two_ hours to find and then run just 12 tests!

However, if I instead run:
    ./kvm check $(./kvm results | awk '/failed/ {print $1}')
i.e., use one thread (two counting awk) to form a list of failed tests
(this step takes a minute), and then distribute just those tests
across $(KVM_PREFIXES) threads I get:
    kvmrunner 3:57.01:   stats/good/failed/previous=failed: 5
    kvmrunner 3:57.01:   stats/good/passed/previous=failed: 4
    kvmrunner 3:57.01:   total: 9
    kvmrunner 3:57.01:   total/failed: 5
    kvmrunner 3:57.01:   total/passed: 4
which is a whopping 4 minutes to do roughly the same amount of work.

So what gives?  Running <<./kvm results>> on one core takes about 1
minute, not nearly 2 hours which was multi-threaded.
I'm going to point a finger at Python's uber lock.  It is wrapped
around pretty much everything.

On Thu, 1 Jul 2021 at 14:22, Andrew Cagney <andrew.cagney at gmail.com> wrote:
>
> Another might be to try building from /tmp - there's a significant amount of data being written?  vs debug.log
>
> On Thu, 1 Jul 2021 at 14:18, Andrew Cagney <andrew.cagney at gmail.com> wrote:
>>
>> Set KVM_LOCALDIR=/tmp/pool ?
>>
>> On Thu, 1 Jul 2021 at 09:23, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
>>>
>>> I have two similar machines for running tests.
>>> - ThinkCentre M93p Tiny
>>> - i5-4570T CPU @ 2.90GHz (Haswell generation)
>>> - as far as I know, all available firmware updates for security (i.e.
>>>   slowdowns)
>>> - Fedora 34 (arrived at by many upgrades)
>>>
>>> They differ in:
>>> - the first has a 2.5" HDD and 16G of RAM.
>>> - the second has a SATA SSD and 8G of RAM
>>>
>>> A complete kvm-check took
>>> - on the first  6:10:46 elapsed time
>>> - on the second 5:25:39 elapsed time
>>>
>>> The first system takes about 14% more time than the second.
>>>
>>> I suspect that the majority of the effect was disk speed.
>>> I would have guessed that disk speed would have mattered more.
>>> _______________________________________________
>>> Swan-dev mailing list
>>> Swan-dev at lists.libreswan.org
>>> https://lists.libreswan.org/mailman/listinfo/swan-dev