[Swan-dev] Clearing the deck for the f28 switch ...

[Antony Antony]

To me this cleanup is a good idea, however, it is not a strict necessity for F28 migration!
Anyway lets continue this cleanup, without rushing it.

On Mon, Jul 30, 2018 at 05:55:49PM -0400, Andrew Cagney wrote:
> On Mon, 30 Jul 2018 at 16:38, Paul Wouters <paul at nohats.ca> wrote:
> >
> > On Sat, 28 Jul 2018, Andrew Cagney wrote:
> >
> > >> Please don’t.
> > >>
> > >> Some tests were put in wip when they shouldn’t have been are were neglected as a result.
> > >
> > > I don't follow.
> > >
> > > Surely any test that consistently fails is going to get ignored.
> >
> > But less so when they dont appear anywhere whatsoever or are deleted.
> > Some tests were moved to WIP against my wishes :P
> 
> I'll post the list in a new thread.
> 
> > > Where the problem lies is when those failing tests are mixed in with
> > > tests that should pass.  Firtly, the test process is made needlessly
> > > slower, secondly, they add even more noise to the results making it
> > > even harder to find identify real failures.
> >
> > I agree, but it all really depends on why the test is WIP.
> >
> > >> We should only delete tests after confirming they don’t test anything useful anymore.
> > >>
> > >> Some tests as wip pending code and should not be deleted
> > >
> > > I've not suggested deleting wip tests.  Just doing something with
> > > tests that do not even appear in TESTLIST.
> >
> > Those that do not appear on TESTLIST should be checked why not. So who
> > put them in, do they have a directory they forgot to git add ?
> >
> > I'm okay with notifying test owners of missing tests they have 1-2 weeks
> > to fix or the TESTLIST entry will be removed.

going forward we could be more aggressive about these. I will pay more attention.

> I'm talking about the reverse.  What looks like a directory under
> testing/pluto/ but with NO corresponding TESTLIST entry.

some of them are (important) experiments. Such as namespace test, or docker,
Crypto overload tests,..
It is in the testing directory to able to run manually, with swantest.

I noticed, VERY shortly after you send the first e-mail, the outputs of such tests are removed. I like to leave the outputs there.
e.g
testing/pluto/ikev2-37-docker-rw/east.console.txt
testing/pluto/ikev2-37-docker-rw/road.console.txt

Paul mentioned he might revert a689326169?  Are you goig to do that?

Andrew could you explain what is your issue with output such as ikev2-37-docker-rw/road.console.txt? And how exactly it is related to F28?

I can imagine some inconvenience while updating outputs blindly, such as with sed.
However, I am of the opinion do not update blindly. Only update if the test ran.
If it is not in the TESTLIST it won't run and don't update.
Some tests didn't run sanitizers. Blind replace on these tests could end up with a very inconstant output and lot of git comments saying "fixed" output.. 

I noticed I forgot to add a couple of them to the TESTLIST.
certoe-12-nat-server. I will rerun them and add soon. Again I don't see this as a blocking issue for f28. We have enough there:)

Some tests in a689326169 seems to be created by you. Did you forgot to add them to the TESTLIST

I suggest send a list with testname and who created it, and date.
git log --pretty="format:%an %ad" ikev2-03-basic-rawrsa-ckaid  | tail -1
Andrew Cagney Mon Jun 20 14:49:42 2016 -0400 

-antony