[Swan-dev] testing: confused by OUTPUT directory fate

D. Hugh Redelmeier hugh at mimosa.com
Sat Jul 25 18:26:37 EEST 2015

| From: D. Hugh Redelmeier <hugh at mimosa.com>

| testing/pluto/netkey-passthrough-03/OUTPUT/RESULT:
| {"hostname": "west", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
| {"hostname": "east", "time": "2015-07-24 17:08", "runtime": 51.0, "msg": "end", "testname": "netkey-passthrough-03", "epoch": 1437772082}
| {"time": "2015-07-24 17:08", "runtime": 56.67, "testname": "netkey-passthrough-03", "epoch": 1437772087, "expect": "good", "node": "bluebird.mimosa.com", "result": "passed"}
| and
| ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/netkey-passthrough-03/OUTPUT/RESULT:
| {"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "west", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.64}
| {"testname": "netkey-passthrough-03", "epoch": 1434833680, "hostname": "east", "time": "2015-06-20 16:54", "msg": "end", "runtime": 47.66}
| {"testname": "netkey-passthrough-03", "epoch": 1434833683, "result": null, "node": "bluebird.mimosa.com", "time": "2015-06-20 16:54", "expect": "good", "runtime": 50.94}

pluto-testlist-scan.sh looks at the RESULT file to determine how to
report whether the tests run succeeded.  In particular, it looks at
the "result" component.  It could not understand ANY of the RESULT
files under ~/result.  How come?

Most that I looked at had "result": null in them.  How the heck does
that happen?

The fact that these two RESULT files look so different suggest that
they are not the same animal.  Surely they should be given different

I did
	cd ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/
	ls -ltr */OUTPUT/RESULT
278 of 312 files were from before 2015-07-24!  Why would that be the case?

When I did
	cd ~/libreswan/testing/pluto/
	ls -ltr */OUTPUT/RESULT
all 197 files were from 2015-07-24.  197 has no obvious relation to
278 and 312.  Huh???

Clearly at least some of the RESULT files in ~/libreswan are more up
to date than the corresponding RESULT file in ~/result (197 > 312-278).

Another probe:
	cd ~/results/bluebird.mimosa.com/2015-07-24-bluebird.mimosa.com-v3.14rc3-66-g28f5374-master/
	grep -L '"result": null' */OUTPUT/RESULT | wc -l
This prints 35.  So all but 35 RESULT files have a "result": null.
That's a tremendous loss of data, data that takes a lot of time to gather.
278 + 35 == 313.  Close to 312.

I don't mind graphical presentations that I don't yet use.  I do mind
results being thrown away (if that is in fact happening).

This is all too mysterious.  It is getting in the way of my testing.

More information about the Swan-dev mailing list