<div dir="ltr">Thank you Paul.<div><br></div><div>I added "compress=yes", seems not help. Compression_algorithm is only used in phase 2, right? The failure here is phase 1. Firewall does not like the issue either because packet on UDP 4500 can be seen on both end.</div><div><br></div><div>I have attached racoon log file. I see:</div><div>
<p class="gmail-p1"><span class="gmail-s1">invalid length of payload<br>
</span></p><p class="gmail-p1"><span class="gmail-s1">ERROR: phase1 negotiation failed due to time up. 8c6688cabaaf90ca:a8d8758b59948609</span></p><p class="gmail-p1"><span class="gmail-s1">
</span></p><p class="gmail-p1">ERROR: phase2 negotiation failed due to time up waiting for phase1. ESP 10.2.128.240[0]->10.0.0.1[0]</p><p class="gmail-p1"><br></p><p class="gmail-p1">Thanks,<br>Xinwei</p></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 31, 2017 at 1:53 PM, Paul Wouters <span dir="ltr"><<a href="mailto:paul@nohats.ca" target="_blank">paul@nohats.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Thu, 30 Mar 2017, Xinwei Hong wrote:<br>
<br>
[moderator note: please next time post a pointer to a "pastebin" of the<br>
log instead of included large log files]<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have a VPN setup between libreswan (pluto+netkey) and a racoon (racoon+netkey), the racoon is behind a NAT device. The negotiation somehow failed saying that "NAT-D payload #0 doesn't<br>
match"<br>
</blockquote>
<br></span>
There is not a NAT payload error. Your device has more then one IP<br>
address, and it is trying to calculate the payload for both your<br>
IP addresses. It notes that 10.0.0.1 does not match. But it also<br>
notes that 10.2.128.240 works fine.<br>
<br>
Your actual problem is:<br>
<br>
Mar 30 19:47:02 vvr-10 pluto[24271]: vvr-0:<br>
"conn_vvr-0-ipsectunnel-0-remo<wbr>te-0" #1246: max number of retransmissions<br>
(8) reached STATE_MAIN_I3. Possible authentication failure: no<br>
acceptable response to our first encrypted message<br>
<br>
Your racoon logs will likely have more information in it then your<br>
libreswan log, as racoon is the party rejecting the packet.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
conn conn_vvr-0-ipsectunnel-0<br>
authby=secret<br>
left=10.2.128.240<br>
right=10.2.128.241<br>
ike=3des-sha1;modp1024<br>
phase2alg=3des-sha1;modp1024<br></span>
leftsubnet=<a href="http://10.100.0.0/24" rel="noreferrer" target="_blank">10.100.0.0/24</a><br>
rightsubnet=<a href="http://10.100.1.0/24" rel="noreferrer" target="_blank">10.100.1.0/24</a><br>
</blockquote><span class="">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
on racoon, we have racoon.conf<br>
# Phase 1 (Main Mode) Configuration<br>
remote 10.2.128.240 {<br>
exchange_mode main;<br>
proposal_check obey;<br>
lifetime time 28800 seconds;<br>
nat_traversal on;<br>
#script "phase1-up.sh" phase1_up;<br>
#script "phase1-down.sh" phase1_down;<br>
dpd_delay 15; dpd_retry 5; dpd_maxfail 5;<br>
proposal {<br>
encryption_algorithm 3des;<br>
hash_algorithm sha1;<br>
dh_group modp1024;<br>
authentication_method pre_shared_key;<br>
}<br>
}<br>
<br>
# Phase 2 (Quick Mode) Configuration/Proposal (for IPsec SA).<br>
sainfo anonymous {<br>
encryption_algorithm 3des;<br>
authentication_algorithm hmac_sha1;<br>
pfs_group modp1024;<br>
lifetime time 3600 seconds;<br>
compression_algorithm deflate;<br>
}<br>
</blockquote>
<br></span>
The only mismatch I see is for compression. You can try and removing the<br>
compression line from racoon, or adding compress=yes to libreswan.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Mar 30 19:47:52 testhost-601-1 racoon: ERROR: phase1 negotiation failed due to time up. 80b77211a2f1ddba:141872152ca77<wbr>72f<br>
</blockquote>
<br></span>
This is odd. Another possibility is that you have firewalled UDP 4500<br>
and so the NAT switching from UDP 500 to UDP 4500 is failing?<span class="HOEnZb"><font color="#888888"><br>
<br>
Paul<br>
</font></span></blockquote></div><br></div></div>