[Swan-dev] Clearing the deck for the f28 switch ...

Andrew Cagney andrew.cagney at gmail.com
Wed Aug 1 18:23:32 UTC 2018


I've posted two lists:

- tests that just don't pass and should be moved to WIP
- test directories with no matching TESTLIST entry

some more notes inline

> To me this cleanup is a good idea, however, it is not a strict necessity for F28 migration!
> Anyway lets continue this cleanup, without rushing it.

> > > Those that do not appear on TESTLIST should be checked why not. So who
> > > put them in, do they have a directory they forgot to git add ?

All the above.  The process is remarkably error-proned.  We're all
making mistakes.

For a TESTLIST entry that has no directory, the kvm tools do print an error:
     kvmresults 0.02: directory 'testing/pluto/...' is invalid
However, I don't think it reduced the error rate.  Instead it is
serving as an alert after-the-event.

For directory with no TESTLIST entry we don't even get an error (I
just pushed a script to at least find these).

My personal preference is to dump TESTLIST and instead run anything
that looks like a test directory.  To me the root cause of these
problems is having to maintain the information in two places.

Another option would be to hack the kvm tools so that they do more
cross checking.

> > > I'm okay with notifying test owners of missing tests they have 1-2 weeks
> > > to fix or the TESTLIST entry will be removed.
>
> going forward we could be more aggressive about these. I will pay more attention.
>
> > I'm talking about the reverse.  What looks like a directory under
> > testing/pluto/ but with NO corresponding TESTLIST entry.
>
> some of them are (important) experiments. Such as namespace test, or docker,
> Crypto overload tests,..
> It is in the testing directory to able to run manually, with swantest.

Can these be included under skiptest?

#   skiptest: used for unit tests where a human is expected to examine
#             the output; this test is not run by default
kvmplutotest    ikev2-delete-03-valgrind        skiptest
kvmplutotest    ikev2-56-restart        skiptest
kvmplutotest    basic-pluto-01-valgrind    skiptest
....

this way we know that these are not a mistake.

> I noticed, VERY shortly after you send the first e-mail, the outputs of such tests are removed. I like to leave the outputs there.

This is what I wrote in my post that started this thread.  Not exactly
clandestine:

- delete tests completely missing from TESTLIST:
I've already removed those test's expected output as it just gets in
the way of mass updates.  The input side is similar.


More information about the Swan-dev mailing list