<div dir="ltr"><br><div class="gmail_extra"><br><div>On 26 May 2015 at 12:38, D. Hugh Redelmeier <span dir="ltr"><<a href="mailto:hugh@mimosa.com" target="_blank">hugh@mimosa.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">| From: Andrew Cagney <<a href="mailto:andrew.cagney@gmail.com" target="_blank">andrew.cagney@gmail.com</a>><br>
<span><br>
| re-sanitize.sh and pluto-testlist-scan.sh are nice compact scripts, each<br>
| with a singular job.<br>
| It might be better to just keep them separate (on the other hand, having<br>
| the decision that a test passed in multiple places isn't good).<br>
<br>
</span>pluto-testlist-scan.sh was born out of my confusion. I could not<br>
figure out how to efficiently figure out how the tests were doing.<br>
<br></blockquote><div><br></div><div>That and your points below are insightful.. I was wondering if the scripts could be simplified to always perform sanitization, clearly not.<br><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
It is meant to be useful in the middle of running the suite as well as<br>
after completion. Waiting 12 hours for a simple "you blew it" is<br>
wasteful.<br>
<br></blockquote><div><br>For my part, and for possibly similar reasons I've come to bypass the
RESULT and table.txt files and instead examine the diff files directly (also comparing their contents against earlier baselines).<br><br></div><div>I ignore RESULT files because:<br><br>- a RESULT file indicating a failure has no helpful context, that is found in the diff; consequently I just look at diffs<br><br>- I can't compare RESULT files using a simple diff - this is because they contain statistical information that changes across runs<br><br></div><div>- the json format is harder for me, as a human, to parse; its also harder to process using traditional UNIX scripting<br></div><div><br></div><div>On the other hand, I find the diff files easy to manipulate:</div><div><br></div><div>- an empty diff indicates (for the most part) things are ok<br><br></div><div>- a non-empty diff file can be quickly examined and compared (against a baseline) to determine if it really is a failure<br><br></div><div>Taking a step back, perhaps what we need to do is separate the statistical information (consumed by the test infrastructure) from the low-level test results. For instance:<br><br>- OUTPUT/result.txt: If present the test ran. If empty the test passed. Otherwise it contains the concatenation of all the diffs along with any fatal errors extracted from the logs.<br><br></div><div>- OUTPUT/RESULT: a json file containing those useful statistics along with a summary extracted from result.txt<br></div><br><br> <br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I would be happy if it were obsolete. I don't know whether it is.<br>
<br>
I do think that the GUI-based tool is way slicker (in a good way) than<br>
pluto-testlist-scan.sh and shows promise. But:<br>
<br>
- It doesn't work out of the box (we don't distribute necessary<br>
parts).<br>
<br>
- It works in ways counter to my intuition (perhaps documentation<br>
could address this).<br>
<br>
- It requires machinery that I don't really want to run on my test<br>
machine.<br>
<br>
- As far as I know, it only works after the test suite run completes.<br>
<br></blockquote><div><br></div><div>I've dusted off some Python code I've had laying around here and created testing/utils/kvmresults.py. It lists each test along with pass / fail / not-started / not-finished. It just looks at .diff files (not RESULT). Depending on how this discussion goes it could be tweaked to include all the functionality provided by pluto-testlist-scan.sh.<br><br></div><div>For the moment, though, it provides an interesting counter example: a, probably naive, implementation of something like pluto-testlist-scan.sh in python.<br></div><div><br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
(Of course I could be wrong about any of these points. I haven't even<br>
gotten it running yet. I'm trying, and with Paul's help, I'm close.)<br>
<br>
pluto-testlist-scan.sh isn't documented either. At least you can read<br>
the code easily. It's only a stop-gap.<br></blockquote><div><br></div><div>Andrew<br><br><br></div></div></div></div>