[Swan-dev] testing: drifting reference logs

[Paul Wouters]

On Fri, 2 Jan 2015, Antony Antony wrote:

> We have another output change that affect almost all test cases(200+) now.
>
> -000 algorithm IKE encrypt: v1id=13, v1name=DISABLED-OAKLEY_AES_CTR, v2id=13, v2name=AES_CTR, blocksize=16, keydeflen=128
> +000 algorithm IKE encrypt: v1id=8, v1name=OAKLEY_CAMELLIA_CBC, v2id=23, v2name=CAMELLIA_CBC, blocksize=16, keydeflen=128

It should be more than just these two lines, as both OAKLEY_AES_CTR and
OAKLEY_CAMELLIA_CBC got enabled recently.

> I suggest output update only if the test pass.  That would increase the quality.

I'm working through a full test run still after I merged in the CTR code
from Andrew.

> My feeling about WIP test cases is do not commit output.

I guess we all still disagree on the best method...... I've changed the
output but will do a full testrun before commiting the updated output.

> Paul if you are going to convert all kvm test cases, first make output zero byte file. And only after they pass, preferably on more than one machine, commit the output. Otherwise, from the past 6 months of experience, it feels like a lot of work to maintain them.

I just removed all console output files from tests marked as umltest
(some exceptions for some ipv6 tests which i know were converted but
was not marked as kvmplutotest)

If the full test run goes well, tonight the tests should be passing
again. Sorry for the noise,

Paul