[Swan] communication failures during high traffic volume

[dsnail at email.com]

We have a configuration (RHEL 6.8,  Libreswan 3.15) where one node (sgltmgw) is passing incoming HTTP traffic to HTTP servers (i.e. sgdmail12).  The nodes communicate only through IPSec tunnels.   When we try to push 1000 or more http req/sec and 200 Mbps traffic through libreswan,  we begin to see failures.  We've seen no network or node resource issues or errors at the time of the failures.  This is reproducible every time and at lower traffic volume there is no problem.   

Our configurations look similar to the following:

conn sgdmail12-to-sgltmgw-on-any
    leftid=%fromcert
    left=A.B.183.116
    rightid=%fromcert
    right=A.B.182.110
    rightrsasigkey=%cert
    ike=aes-sha2_256-modp1536
    phase2alg=aes_gcm_c-128-null
    rightcert=sgdmail12.YYY.com
    rightsendcert=always
    authby=rsasig
    auto=start
    failureshunt=drop

conn sgltmgw-to-sgdmail12-on-any
    leftid=%fromcert
    left=A.B.183.116
    leftrsasigkey=%cert
    rightid=%fromcert
    right=A.B.182.110
    ike=aes-sha2_256-modp1536
    phase2alg=aes_gcm_c-128-null
    leftcert=sgltmgw.YYY.com
    leftsendcert=always
    authby=rsasig
    auto=start
    failureshunt=drop

I've attached two partial logs.  A failure time frame is Feb 3 20:00 till 20:11.  We see messages like "phase 1 message is part of an unknown exchange" (20:05:44) .  If we let it continue eventually see messages like "max number of retransmissions (8) reached STATE_MAIN_R1" (20:06:47). We've seen no network or node resource issues or errors at the time of the failures.   We turned off DPD because Libreswan was declaring nodes down during similar testing.  So we turned on debug-all.  Do the logs have any explanation for this behavior?  


-------------- next part --------------
A non-text attachment was scrubbed...
Name: sgdltmgwplutosmall.txt.gz
Type: application/octet-stream
Size: 99543 bytes
Desc: not available
URL: <https://lists.libreswan.org/pipermail/swan/attachments/20170203/a4872109/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sgdmail12pluto.txt.gz
Type: application/octet-stream
Size: 15303 bytes
Desc: not available
URL: <https://lists.libreswan.org/pipermail/swan/attachments/20170203/a4872109/attachment-0003.obj>