[Swan-dev] testing: confused by OUTPUT directory fate

[Andrew Cagney]

Having played with this a little, I've concluded that the test runner
should look at OUTPUT and not .../results when deciding if a test
should be run; and probably only delete OUTPUT when UPDATE=1

It's much clearer.

Andrew



On 25 July 2015 at 03:35, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> As mentioned earlier, my test run from last night hung in one test.
> I had initiated with
>         make check UPDATE=1
>
> I killed the hung test run (it took a couple of ^C strokes to do it).  I
> then restarted it via
>         make check
>
> Somehow all the testing/pluto/*/OUTPUT directories that had been
> painstakingly filled in the first run disappeared.  I'm not used to this,
> but it is the way things work now.  They were probably moved to
> ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/*/OUTPUT/
>
> The oldest testing/pluto/*/OUTPUT director after the second run is
> drwxrwxr-x. 2 build build 4096 Jul 24 12:45 testing/pluto/ikev2-ddns-02/OUTPUT
> This directory does not appear under results.  What's that about?
>
> And it doesn't appear in the expected sequence as per TESTLIST
> The next oldest is
> drwxrwxr-x. 2 build build 4096 Jul 24 12:49 testing/pluto/newoe-07-ike-replace-initiator/OUTPUT
> Why is that?
>
> In order to salvage the run, I have changed my copy of
> pluto-testlist-scan.sh to be willing to look in ~/results for the OUTPUT directories.
> But it fails.  Why?
>
> Let's look at
>
> testing/pluto/netkey-passthrough-03/OUTPUT/RESULT:
>
> {"hostname": "west", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
> {"hostname": "east", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
> {"time": "2015-07-24 17:08", "runtime": 56.67, "testname": "netkey-passthrough-03", "epoch": 1437772087, "expect": "good", "node": "bluebird.mimosa.com", "result": "passed"}
>
> and
> ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/netkey-passthrough-03/OUTPUT/RESULT:
>
> {"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "west", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.64}
> {"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "east", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.66}
> {"testname": "netkey-passthrough-03", "epoch": 1434833683, "result": null, "node": "bluebird.mimosa.com", "time": "2015-06-20 16:54", "expect": "good", "runtime": 50.94}
>
> They ought to be the same since they describe the same test run (I think).
> They are quite different.  Why the heck is that?
>
> In fact, the one in ~/results seems to be from a different date!
> $ ls -ltr ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/netkey-passthrough-03/OUTPUT/
> total 644
> -rw-rw-r--. 1 build build      0 Jun 20 16:53 swan12.pcap
> -rw-rw-r--. 1 build build  20210 Jun 20 16:54 west.console.verbose.txt
> -rw-rw-r--. 1 build build  16617 Jun 20 16:54 east.console.verbose.txt
> -rw-rw-r--. 1 build build    448 Jun 20 16:54 RESULT
> -rwxrwxrwx. 1 build qemu  288592 Jul 24 17:08 east.pluto.log
> -rwxrwxrwx. 1 build qemu  308696 Jul 24 17:08 west.pluto.log
> -rw-rw-r--. 1 build build   3789 Jul 24 17:08 east.console.txt
> -rw-rw-r--. 1 build build      0 Jul 24 17:08 east.console.diff
> -rw-rw-r--. 1 build build   6684 Jul 24 17:08 west.console.txt
> -rw-rw-r--. 1 build build      0 Jul 24 17:08 west.console.diff
>
> Why the heck are things from June 20 here?  After all, the directory
> name includes the date July 24.
>
> [build at bluebird libreswan]$ ls -ltr testing/pluto/netkey-passthrough-03/OUTPUT/
> total 644
> -rw-rw-r--. 1 build build  20209 Jul 24 17:08 west.console.verbose.txt
> -rw-rw-r--. 1 build build  16616 Jul 24 17:08 east.console.verbose.txt
> -rwxrwxrwx. 1 build qemu  288592 Jul 24 17:08 east.pluto.log
> -rwxrwxrwx. 1 build qemu  308696 Jul 24 17:08 west.pluto.log
> -rw-rw-r--. 1 build build   6684 Jul 24 17:08 west.console.txt
> -rw-rw-r--. 1 build build      0 Jul 24 17:08 west.console.diff
> -rw-rw-r--. 1 build build   3789 Jul 24 17:08 east.console.txt
> -rw-rw-r--. 1 build build      0 Jul 24 17:08 east.console.diff
> -rw-rw-r--. 1 build build    450 Jul 24 17:08 RESULT
>
> The are NOT in the original directory.
>
> This looks very very wrong.
>
> Can anyone explain?
>
> I was going to start another test run, but I think that I need to
> leave the scene of the crime undisturbed.
> _______________________________________________
> Swan-dev mailing list
> Swan-dev at lists.libreswan.org
> https://lists.libreswan.org/mailman/listinfo/swan-dev