[Swan-dev] understanding output of the tests

[Paul Wouters]

On Wed, 2 Apr 2014, D. Hugh Redelmeier wrote:

> As a newbie, I find the test output unfamiliar and hard to make sense of.
>
> Each test has a file descriptions.txt which is very helpful (of course they
> could not be improved).

And I would like those descriptions to become more json friendly using
keywords.

> Some prompts and commands don't seem to line up properly.  This would
> seem to be a challenge to the integrity of the tests.  Here's an
> example from my current run of basic-pluto-01:
>
> west #
> - ipsec auto --up  westnet-eastnet
> + #ipsec auto --up  westnet-eastnet
> +west #
> + ipsec whack --name westnet-eastnet --initiate
> +002 "westnet-eastnet" #1: initiating Main Mode
> 104 "westnet-eastnet" #1: STATE_MAIN_I1: initiate
>
> BTW, how did whack appear?  Was the script changed?

Sometimes there is a flow control problem with expect/pexpect :(
Repeating the run usually fixes it. This is one reason why we
NEVER have blanc lines in our .sh scripts, those seem to have
a much higher chance of triggering these.

> ================
>
> BTW, there's a lot of AVC crud in my logs (I run with selinux in
> Permissive mode).

Yes. Because we are using non-standard locations like /testing and
/source and /usr/local. We tried to fix some of those using mount
options (see /etc/fstab within the guest). But we need some custom
policy that I was hoping Tuomo would work on.

> ================
>
> In basic_pluto_02, the description.txt leaves me pretty confused about
> what is supposed to happen.  I get a diff and cannot tell if it is
> actually more or less correct than the reference logs.

Yes, unfortunately, the descriptions cannot always be trusted as
sometimes they have been copied from other testcases and forgotten
to be updated.

> On the other hand, my log ends without some commands that appear in
> the original.  How can that happen?  It looks as if only half of the
> the westrun.sh script was run.

It can be that something was updated globally (in all test cases) but
the new run hasn't actually been run/commited yet.

> How can I diagnose this?  The shell ended without anything showing up
> in the log.
>
> There isn't any conn loaded on east, I think.

If no conns loaded, I would ssh root at east and manually load the conn and
check the error.

> ================
>
> westrun.sh ends with:
> 	echo done
> Surely something like this would be better:
> 	echo "done westrun.sh"

It's just a marker for us. I don't think it really matters what filename
it is, and I wouldnt want to need to change it in the future if it
changes. Also image converting the testcase to a roadwarrior one where
we use "road" instead of "west". We are always trying to make the
commands and output as generic as possible allowing easy "copy and
change" of test cases. For instance, the old uml ones had the test case
exported in th init.sh scripts, meaning you always had to edit those.

> It would let the reader get his bearings.  It would be a better anchor
> for diffs too.

I don't think so, the diffs of west are in OUTPUT/west.console.diff
anyway. There is no confusion with east here.

> ================
>
> Since shell prompts are #, perhaps the scripts fed to expect should
> use : for shell comments.  Then the two uses won't accidentally match
> up.

> Commenting using : is done some places in the scripts.

That's a good idea. We have been a little inconsistent with those, but
switching all to : could help detect flow issues.

> : for comments has a few risks.  Put ' ' around the body for safety.
>
> 	# ==== end ====
> becomes
> 	: '==== end ===='

Why are the quotes needed when using : ?

> ================
>
> dpd-01 and dpd-01-netkey don't make particular sense to me.  I cannot
> tell if my system passed

There are lots of comments for that one. We basically check the tunnel
goes from established to trap to established.

> ================
>
> pluto/algo-pluto-02: my run went on a lot longer than the reference.

Note that testcase still has the old style "west-console.txt" instead of
"west.console.txt". It has not been fully converted yet. You can also
see it still uses the ipsec auto --status in the init.sh files, which we
have removed from almost all test cases because it gives too many false
positives due to new features in the status output. You can also see
it states "# confirm with a ping to east-in" which is a sign I haven't
changed it fully. "east-in" used to be a name that was pingable, but
isn't anymor, and I change the text to "# confirm with a ping". I'll
update that test now.

Paul