[Swan-dev] testing: confused by OUTPUT directory fate

[D. Hugh Redelmeier]

As mentioned earlier, my test run from last night hung in one test.
I had initiated with
	make check UPDATE=1

I killed the hung test run (it took a couple of ^C strokes to do it).  I 
then restarted it via
	make check

Somehow all the testing/pluto/*/OUTPUT directories that had been 
painstakingly filled in the first run disappeared.  I'm not used to this, 
but it is the way things work now.  They were probably moved to
~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/*/OUTPUT/

The oldest testing/pluto/*/OUTPUT director after the second run is
drwxrwxr-x. 2 build build 4096 Jul 24 12:45 testing/pluto/ikev2-ddns-02/OUTPUT
This directory does not appear under results.  What's that about?

And it doesn't appear in the expected sequence as per TESTLIST
The next oldest is
drwxrwxr-x. 2 build build 4096 Jul 24 12:49 testing/pluto/newoe-07-ike-replace-initiator/OUTPUT
Why is that?

In order to salvage the run, I have changed my copy of
pluto-testlist-scan.sh to be willing to look in ~/results for the OUTPUT directories.
But it fails.  Why?

Let's look at

testing/pluto/netkey-passthrough-03/OUTPUT/RESULT:

{"hostname": "west", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
{"hostname": "east", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
{"time": "2015-07-24 17:08", "runtime": 56.67, "testname": "netkey-passthrough-03", "epoch": 1437772087, "expect": "good", "node": "bluebird.mimosa.com", "result": "passed"}

and
~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/netkey-passthrough-03/OUTPUT/RESULT:

{"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "west", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.64}
{"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "east", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.66}
{"testname": "netkey-passthrough-03", "epoch": 1434833683, "result": null, "node": "bluebird.mimosa.com", "time": "2015-06-20 16:54", "expect": "good", "runtime": 50.94}

They ought to be the same since they describe the same test run (I think).
They are quite different.  Why the heck is that?

In fact, the one in ~/results seems to be from a different date!
$ ls -ltr ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/netkey-passthrough-03/OUTPUT/
total 644
-rw-rw-r--. 1 build build      0 Jun 20 16:53 swan12.pcap
-rw-rw-r--. 1 build build  20210 Jun 20 16:54 west.console.verbose.txt
-rw-rw-r--. 1 build build  16617 Jun 20 16:54 east.console.verbose.txt
-rw-rw-r--. 1 build build    448 Jun 20 16:54 RESULT
-rwxrwxrwx. 1 build qemu  288592 Jul 24 17:08 east.pluto.log
-rwxrwxrwx. 1 build qemu  308696 Jul 24 17:08 west.pluto.log
-rw-rw-r--. 1 build build   3789 Jul 24 17:08 east.console.txt
-rw-rw-r--. 1 build build      0 Jul 24 17:08 east.console.diff
-rw-rw-r--. 1 build build   6684 Jul 24 17:08 west.console.txt
-rw-rw-r--. 1 build build      0 Jul 24 17:08 west.console.diff

Why the heck are things from June 20 here?  After all, the directory
name includes the date July 24.

[build at bluebird libreswan]$ ls -ltr testing/pluto/netkey-passthrough-03/OUTPUT/
total 644
-rw-rw-r--. 1 build build  20209 Jul 24 17:08 west.console.verbose.txt
-rw-rw-r--. 1 build build  16616 Jul 24 17:08 east.console.verbose.txt
-rwxrwxrwx. 1 build qemu  288592 Jul 24 17:08 east.pluto.log
-rwxrwxrwx. 1 build qemu  308696 Jul 24 17:08 west.pluto.log
-rw-rw-r--. 1 build build   6684 Jul 24 17:08 west.console.txt
-rw-rw-r--. 1 build build      0 Jul 24 17:08 west.console.diff
-rw-rw-r--. 1 build build   3789 Jul 24 17:08 east.console.txt
-rw-rw-r--. 1 build build      0 Jul 24 17:08 east.console.diff
-rw-rw-r--. 1 build build    450 Jul 24 17:08 RESULT

The are NOT in the original directory.

This looks very very wrong.

Can anyone explain?

I was going to start another test run, but I think that I need to
leave the scene of the crime undisturbed.