[Swan-dev] confused by make check [was: ikev2-frag-02-ipv6 fails due to certificate problems]

[D. Hugh Redelmeier]

| From: D. Hugh Redelmeier <hugh at mimosa.com>

| People who understand our makefiles: please read.

Please, please.

| ...  Now I wanted to stop the tests that so I
| could genererate the testing certificates.  I did
| 	touch ~/libreswan/testing/pluto/stop-tests-now
| following the chart
| 	<https://libreswan.org/wiki/Test_Suite#stopping_pluto_tests_gracefully>
| 
| ===> THIS DID NOT WORK
| 
| It went through all the post-test rigamarole (lots of exciting greps
| and so on, scrolling off my screen).  And then just continued running
| tests!  So that mechanism seems broken.
| ~/libreswan/testing/pluto/stop-tests-now is still there.

It is way worse than that.

It went on for hours, running lots of tests.  But it lost their
output!

I saved the host console output from the two make check runs:
	make check UPDATE=1 <=== interrupted by power failure
	make check <=== to resume
The second one was partially interrupted by the touch command.

The following comments are based on the log of the seconde make check.

This message appeared 255 times:
	****** skip tests found stop-tests-now in dir /home/build/libreswan/testing/pluto *****
Clearly the stop file was not ignored.  But it didn't stop the train.

Many messages appear seven times.  For example:
      ****** invalid test xauth-pluto-17: directory /home/build/results/bluebird.mimosa.com/2015-05-24-bluebird.mimosa.com-v3.13oe6-190-ga8c95e1-dirty-oe-scratch3/xauth-pluto-17 not found
This makes NO sense.  Why is it looking at a run from a month ago?!?
Seven times!

A bunch of messages appear four times.  For example:
      ****** skipping test xauth-pluto-19: /home/build/results/bluebird.mimosa.com/2015-06-20-bluebird.mimosa.com-v3.14rc2-32-gb53528a-master/xauth-pluto-19/OUTPUT/RESULT exists and max retry count 5 exceeded
This makes more sense since it is about today's testing.
What's with this retry count of 5?!?  What's going on?

A bunch of messages appear three times.  For example:
      ****** skipping test netkey-audit-01: /home/build/results/bluebird.mimosa.com/2015-06-20-bluebird.mimosa.com-v3.14rc2-32-gb53528a-master/netkey-audit-01/OUTPUT/RESULT exists, do not run again
This too is actually about this latest run (good).
What's it saying?  Is it worth saying three times?
Some similar messages appear only twice.

A bunch of messages appear twice in.  For example:
      ****** invalid test xauth-pluto-17: directory /home/build/results/bluebird.mimosa.com/2015-06-20-bluebird.mimosa.com-v3.14rc2-32-gb53528a-master/xauth-pluto-17 not found
Seventeen of these pairs refer to this xauth-pluto-17, each from a
different date.  Why? TESTLIST lists it as a skiptest.  Can't we
ignore it once (rather than 34 times)?

A similar set of messages are generated for xauth-pluto-11 which is a
umlXhost test.  And so on for several others.

===> These scripts waste a lot of time and space examining old runs.
     To what purpose?

Good news: tests that were actually run seem to have only been run
once.

Bad news: most OUTPUT directories are missing in action
Some that exist don't have any console logs!

Every test after a certain point (perhaps after I touched the stopfile)
ended with this:
  sanitizer output /bin/sh: ../../utils/sanitize.sh: No such file or directory
  copyig results 
sanitize.sh exists where it is expected to be.

Summary: somehow make check is badly broken.  It needs fixing.