[Swan] libreswan/racoon interoperability problem with NAT-T

[Paul Wouters]

On Thu, 30 Mar 2017, Xinwei Hong wrote:

[moderator note: please next time post a pointer to a "pastebin" of the
log instead of included large log files]

> I have a VPN setup between libreswan (pluto+netkey) and a racoon (racoon+netkey), the racoon is behind a NAT device. The negotiation somehow failed saying that "NAT-D payload #0 doesn't
> match"

There is not a NAT payload error. Your device has more then one IP
address, and it is trying to calculate the payload for both your
IP addresses. It notes that 10.0.0.1 does not match. But it also
notes that 10.2.128.240 works fine.

Your actual problem is:

 	Mar 30 19:47:02 vvr-10 pluto[24271]: vvr-0:
 	"conn_vvr-0-ipsectunnel-0-remote-0" #1246: max number of retransmissions
 	(8) reached STATE_MAIN_I3.  Possible authentication failure: no
 	acceptable response to our first encrypted message

Your racoon logs will likely have more information in it then your
libreswan log, as racoon is the party rejecting the packet.

> conn conn_vvr-0-ipsectunnel-0
>         authby=secret
>         left=10.2.128.240
>         right=10.2.128.241
>         ike=3des-sha1;modp1024
>         phase2alg=3des-sha1;modp1024
>         leftsubnet=10.100.0.0/24
>         rightsubnet=10.100.1.0/24

> on racoon, we have racoon.conf
> # Phase 1 (Main Mode) Configuration
> remote 10.2.128.240 {
>  exchange_mode main;
>  proposal_check obey;
>  lifetime time 28800 seconds;
>  nat_traversal on;
>  #script "phase1-up.sh" phase1_up;
>  #script "phase1-down.sh" phase1_down;
>  dpd_delay 15; dpd_retry 5; dpd_maxfail 5;
>  proposal {
>   encryption_algorithm 3des;
>   hash_algorithm sha1;
>   dh_group modp1024;
>   authentication_method pre_shared_key;
>  }
> }
> 
> # Phase 2 (Quick Mode) Configuration/Proposal (for IPsec SA).
> sainfo anonymous {
>  encryption_algorithm 3des;
>  authentication_algorithm hmac_sha1;
>  pfs_group modp1024;
>  lifetime time 3600 seconds;
>  compression_algorithm deflate;
> }

The only mismatch I see is for compression. You can try and removing the
compression line from racoon, or adding compress=yes to libreswan.

> Mar 30 19:47:52 testhost-601-1 racoon: ERROR: phase1 negotiation failed due to time up. 80b77211a2f1ddba:141872152ca7772f

This is odd. Another possibility is that you have firewalled UDP 4500
and so the NAT switching from UDP 500 to UDP 4500 is failing?

Paul