[Swan-dev] sanitize.sh vs re-sanitize.sh (vs swantest.sh)

Andrew Cagney andrew.cagney at gmail.com
Wed May 27 19:58:50 EEST 2015

On 26 May 2015 at 12:38, D. Hugh Redelmeier <hugh at mimosa.com> wrote:

> | From: Andrew Cagney <andrew.cagney at gmail.com>
> | re-sanitize.sh and pluto-testlist-scan.sh are nice compact scripts, each
> | with a singular job.
> | It might be better to just keep them separate (on the other hand, having
> | the decision that a test passed in multiple places isn't good).
> pluto-testlist-scan.sh was born out of my confusion.  I could not
> figure out how to efficiently figure out how the tests were doing.
That and your points below are insightful..  I was wondering if the scripts
could be simplified to always perform sanitization, clearly not.

> It is meant to be useful in the middle of running the suite as well as
> after completion.  Waiting 12 hours for a simple "you blew it" is
> wasteful.
For my part, and for possibly similar reasons I've come to bypass the
RESULT and table.txt files and instead examine the diff files directly
(also comparing their contents against earlier baselines).

I ignore RESULT files because:

- a RESULT file indicating a failure has no helpful context, that is found
in the diff; consequently I just look at diffs

- I can't compare RESULT files using a simple diff - this is because they
contain statistical information that changes across runs

- the json format is harder for me, as a human, to parse; its also harder
to process using traditional UNIX scripting

On the other hand, I find the diff files easy to manipulate:

- an empty diff indicates (for the most part) things are ok

- a non-empty diff file can be quickly examined and compared (against a
baseline) to determine if it really is a failure

Taking a step back, perhaps what we need to do is separate the statistical
information (consumed by the test infrastructure) from the low-level test
results.  For instance:

- OUTPUT/result.txt: If present the test ran.  If empty the test passed.
Otherwise it contains the concatenation of all the diffs along with any
fatal errors extracted from the logs.

- OUTPUT/RESULT: a json file containing those useful statistics along with
a summary extracted from result.txt

> I would be happy if it were obsolete.  I don't know whether it is.
> I do think that the GUI-based tool is way slicker (in a good way) than
> pluto-testlist-scan.sh and shows promise.  But:
> - It doesn't work out of the box (we don't distribute necessary
>   parts).
> - It works in ways counter to my intuition (perhaps documentation
>   could address this).
> - It requires machinery that I don't really want to run on my test
>   machine.
> - As far as I know, it only works after the test suite run completes.
I've dusted off some Python code I've had laying around here and created
testing/utils/kvmresults.py.  It lists each test along with pass / fail /
not-started / not-finished.  It just looks at .diff files (not RESULT).
Depending on how this discussion goes it could be tweaked to include all
the functionality provided by pluto-testlist-scan.sh.

For the moment, though, it provides an interesting counter example: a,
probably naive, implementation of something like pluto-testlist-scan.sh in

> (Of course I could be wrong about any of these points.  I haven't even
> gotten it running yet.  I'm trying, and with Paul's help, I'm close.)
> pluto-testlist-scan.sh isn't documented either.  At least you can read
> the code easily.  It's only a stop-gap.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libreswan.org/pipermail/swan-dev/attachments/20150527/b90acc01/attachment.html>

More information about the Swan-dev mailing list