[Swan-dev] make clean is eating my testing/pluto/*/OUTPUT directories

[Andrew Cagney]

(I suspect your output is in /home/build/..... somewhere).

Yes, when to clean OUTPUT/* is a good question, I'll get to it below,
but first some archaeology ...

Looking at d6e46094097690796678d9e498c4006c4b7204b8^ (a.k.a. 1 BC) I see:

make[3]: Entering directory
'/home/cagney/libreswan/testdir/OBJ.linux.x86_64/testing/pluto'
grep -v '^#' /home/cagney/libreswan/testdir/testing/pluto/TESTLIST |
while read testtype name status; \
do\
    rm -rf $name/OUTPUT; \
done

so the intent (since 2005) is been to remove OUTPUT during "clean".
However, looking more carefully at the above output, we see the
recursive "make clean" was run in OBJDIR, so didn't actually do very
much :-/

The change bf1d4249c307eb1a38d6002f2b692bc088dc5505 finally flipped
the switch so that "make clean" descended $(srcdir} and, oops, the
code starts "working".

However, and here's the problem,  "make check", in testing/pluto also
deletes OUTPUT:

ifeq ($(UPDATEONLY),)
        make clean
        ../utils/swantest
        ../utils/swantest --scancwd
endif

While I can see about changing recursive "make clean" to not delete
OUTPUT, I suspect having "make check" delete it anyway isn't the
behaviour you're looking for?


So back to my question, when should ${srcdir}/**/OUTPUT be deleted?


First, it is probably worth noting some of swantest's, er,  features:

- by default, it stores results in /home/build/...
try to think of OUTPUT as a scratch directory; not working? try harder -)

- by default, it is incremental and only runs a test if
/home/build/... suggests that the test failed or wasn't run
this means "make check" "make check", even after deleting OUTPUT,
re-runs just the tests that fail; that is useful
I'm guessing that, by deleting OUTPUT, it is easier to see swantest's progress?
On the other hand, deleting OUTPUT not triggering a test re-run is
really confusing (so confusing that I hacked swantest to spit out the
path to the directory I needed to delete); yes, I know, I need to try
harder at thinking of OUTPUT as a scratch directory

- it uses a heuristic based on git to determine if something changed
and the tests should be re-run from scratch (and a new directory in
/home/build/... should be created)
The heuristic isn't very robust.  For instance, given a fresh checkout
and a common /testing and /source:
    make check  # tests everything
    <fix a bug / do not commit>
    make check  # re-tests everything
    <fix a bug / do not commit>
    make check  # only runs failed tests
and, when using different /testing and /source trees:
    make check  # tests everything
    <update pluto from separate directory>
    make check  # only runs failed tests
(it is worth pointing out that making this "robust" is hard, and
possibly not what is wanted)

With these features in mind, I've been trying alternatives (see
kvm-check), for instance:

- not deleting OUTPUT (least surprise)
Unfortunately this makes it harder to see how far the teastsuite has
progressed; perhaps just delete the directories that failed ...

- a slightly more robust check to detect changes and determine that
all tests need to be re-run
This proved less useful than you might expect; I've found that I
instead want incremental test runs, regardless of what changed, as the
default - it lets me incrementally fix and re-run without re-running
everything

so, what's your preference?

Andrew



On 9 June 2015 at 02:34, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> I have noticed a few times recently that my OUTPUT directories disappear.
>
> This is very disconcerting.  Like now: I've lost what I was working on.
>
> It seems to be "make clean".  This is bad news for me because I use "make
> clean" before I "make base".  It does NOT mean that I want OUTPUT gone.
>
> I can see deleting OUTPUT directories on "make distclean" or whatever
> target is like that.
>
> I suggest "make clean" go back to its old behaviour.  Strongly.
>
> BTW a few OUTPUT directories are not deleted.  Perhaps those are for tests
> that are somehow disabled.  I haven't investigated.
> _______________________________________________
> Swan-dev mailing list
> Swan-dev at lists.libreswan.org
> https://lists.libreswan.org/mailman/listinfo/swan-dev