[Swan] Crash with Libreswan 3.11

[Roel van Meer]

Hi list!

today I've tried to migrate our last Openswan installation to Libreswan, but  
unfortunately Libreswan pluto crashes on me:

pluto[29356]: segfault at 38 ip 00007ffb4084e974 sp 00007fff870fef98 error 4 in libnss3.so[7ffb407fe000+138000]

It seems similar to the problem that Rueben Farrelly reported in this  
thread: https://www.mail-archive.com/swan@lists.libreswan.org/msg00269.html
but unlike Rueben's case, I experience the crash immediately after starting  
up.

I've tried several versions of Libreswan (3.8, 3.9, 3.11) and two versions  
of mozilla-nss (3.13.5, 3.16.5), but the crash is seen in all combinations.

This is a complex setup with 160 PSK and 2 rsasig tunnels, with KLIPS.
I haven't tried to pinpoint the problem yet, but I do have two backtraces of  
debug builds (see below). The system is running Slackware, with kernel 3.4.104.

If I run pluto with full debug the crash doesn't happen (or doesn't happen  
always), so that's not really useful.

Any help would be appreciated. I'm going to try the git version now.


Core was generated by `/usr/libexec/ipsec/pluto --config /etc/ipsec.conf -- 
nofork'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f884c72f974 in pk11_ExitKeyMonitor (symKey=<value optimized out>) at pk11skey.c:29
29      pk11skey.c: No such file or directory.
        in pk11skey.c
(gdb) bt
#0  0x00007f884c72f974 in pk11_ExitKeyMonitor (symKey=<value optimized out>) at pk11skey.c:29
#1  0x00007f884c732784 in PK11_DeriveWithTemplate (baseKey=0x7f884d3370e0, derive=866, param=<value optimized out>, target=912, operation=<value optimized out>, keySize=<value optimized out>, userAttr=<value optimized out>, numAttrs=0, isPerm=0) at pk11skey.c:1542
#2  0x00007f884c7329c1 in PK11_Derive (baseKey=<value optimized out>, derive=<value optimized out>, param=<value optimized out>, target=<value optimized out>, operation=<value optimized out>, keySize=<value optimized out>) at pk11skey.c:1394
#3  0x00007f884cc77c72 in pk11_derive_wrapper_lsw (base=<value optimized out>, mechanism=<value optimized out>, data=..., target=<value optimized out>, operation=<value optimized out>, keySize=0) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/hmac.c:238
#4  0x00007f884ccb4e87 in calc_skeyids_iv (r=0x7f8848bce330) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/crypt_dh.c:791
#5  calc_dh_iv (r=0x7f8848bce330) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/crypt_dh.c:1076
#6  0x00007f884ccb1ad0 in pluto_crypto_helper (helper_fd=<value optimized out>, helpernum=2) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/pluto_crypt.c:273
#7  0x00007f884ccb1d51 in pluto_helper_thread (w=<value optimized out>) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/pluto_crypt.c:896
#8  0x00007f884b112d6b in start_thread () from /lib64/libpthread.so.0
#9  0x00007f884a379b7d in clone () from /lib64/libc.so.6


#0  0x00007ffb4084e974 in pk11_ExitKeyMonitor (symKey=<value optimized out>) at pk11skey.c:29
29      pk11skey.c: No such file or directory.
        in pk11skey.c
(gdb) bt
#0  0x00007ffb4084e974 in pk11_ExitKeyMonitor (symKey=<value optimized out>) at pk11skey.c:29
#1  0x00007ffb40851784 in PK11_DeriveWithTemplate (baseKey=0x7ffb30094aa0, derive=866, param=<value optimized out>, target=868, operation=<value optimized out>, keySize=<value optimized out>, userAttr=<value optimized out>, numAttrs=0, isPerm=0) at pk11skey.c:1542
#2  0x00007ffb408519c1 in PK11_Derive (baseKey=<value optimized out>, derive=<value optimized out>, param=<value optimized out>, target=<value optimized out>, operation=<value optimized out>, keySize=<value optimized out>) at pk11skey.c:1394
#3  0x00007ffb40d96c72 in pk11_derive_wrapper_lsw (base=<value optimized out>, mechanism=<value optimized out>, data=..., target=<value optimized out>, operation=<value optimized out>, keySize=64) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/hmac.c:238
#4  0x00007ffb40d97191 in hmac_init (ctx=0x7fff870ff470, h=0x7ffb41070140, symkey=0x7ffb30094aa0) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/hmac.c:81
#5  0x00007ffb40da3507 in compute_proto_keymat (st=0x7ffb419a5c70, protoid=3 '\003', pi=0x7ffb419a5de0, satypename=0x7ffb40e3ec63 "ESP") at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/ikev1_quick.c:458
#6  0x00007ffb40da4021 in quick_inI1_outR1_cryptotail (md=0x7ffb41886710, r=0x7fff870ff9d0) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/ikev1_quick.c:2484
#7  0x00007ffb40da4519 in quick_inI1_outR1_cryptocontinue2 (pcrc=0x7ffb4198e080, r=0x7fff870ff9d0) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/ikev1_quick.c:2315
#8  0x00007ffb40dd1172 in handle_helper_answer (w=0x7ffb4181dbd0) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/pluto_crypt.c:820
#9  0x00007ffb40dd1dbb in pluto_crypto_helper_response_ready (readfds=0x7fff87100a00) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/pluto_crypt.c:1011
#10 0x00007ffb40d93a33 in call_server () at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/server.c:760
#11 0x00007ffb40d8fba5 in main (argc=4, argv=<value optimized out>) at /tmp/tmp.pCVGT6/libreswan-3.11/programs/pluto/plutomain.c:1387


Thanks!

Roel