[Swan-dev] Pluto memory consumption

[Andrew Cagney]

On 28 February 2017 at 09:28, D. Hugh Redelmeier <hugh at mimosa.com> wrote:
> NSS does its own memory allocation and is thus invisible to the leak
> detective.  Anything NSS-related is thus suspect.  Think: keys and
> related stuff.  So you are probably on the right track.
>
> | ==2935== 5,095,216 bytes in 938 blocks are still reachable in loss record 652
>
> | ==2935==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
> | ==2935==    by 0x6B3B351: PORT_ZAlloc_Util (in /usr/lib64/libnssutil3.so)
>
> | ==2935==    by 0x16B228: symkey_from_symkey (crypt_symkey.c:283)

Yea, a simpler one is:

==2935== 16 bytes in 1 blocks are still reachable in loss record 38 of 653
==2935==    at 0x4C29BE3: malloc (vg_replace_malloc.c:299)
==2935==    by 0x6B3B27F: PORT_Alloc_Util (in /usr/lib64/libnssutil3.so)
==2935==    by 0x4E79E59: ??? (in /usr/lib64/libnss3.so)
==2935==    by 0x4E8428C: PK11_ExtractKeyValue (in /usr/lib64/libnss3.so)
==2935==    by 0x4E84567: ??? (in /usr/lib64/libnss3.so)
==2935==    by 0x4E854DB: PK11_DeriveWithTemplate (in /usr/lib64/libnss3.so)
==2935==    by 0x4E85720: PK11_Derive (in /usr/lib64/libnss3.so)
==2935==    by 0x16B0BF: merge_symkey_bytes (crypt_symkey.c:222)
==2935==    by 0x16BD72: symkey_from_bytes (crypt_symkey.c:376)
==2935==    by 0x1336A1: decode_to_key (test_buffer.c:157)
==2935==    by 0x132811: test_cbc_vector (cbc_test_vectors.c:83)
==2935==    by 0x132811: test_cbc_vectors (cbc_test_vectors.c:111)
==2935==    by 0x13425E: init_crypto (crypto.c:52)

where the code looks like:

    PK11SymKey *sym_key = decode_to_key(encrypt_desc, test->key);
    if (!test_cbc_op(encrypt_desc, test->description, 1,
             sym_key, test->iv,
             "plaintext: ", test->plaintext,
             "ciphertext: ", test->ciphertext)) {
        ok = FALSE;
    }
    if (!test_cbc_op(encrypt_desc, test->description, 0,
             sym_key, test->iv,
             "cipertext: ", test->ciphertext,
             "plaintext: ", test->plaintext)) {
        ok = FALSE;
    }

    /* Clean up.  */
    free_any_symkey("sym_key", &sym_key);

so from our POV the key was freed.  However NSS has kept a handle on
that memory and will recycle it repeatedly.